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Editor’s  Note 

Army  contract  data  item  description  DI-H-1334A, 
entitled  “Report  of  HFE  Test”  is  referred  to  throughout  this 
report.  That  data  item  description  has  recently  been 
republished  as  Department  of  Defense  contract  data  item 
description  DI-H-7058. 
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GUIDE  FOR  OBTAINING  AND  ANALYZING  HUMAN  PERFORMANCE 


DATA  IN  A MATERIEL  DEVELOPMENT  PROJECT 


INTRODUCTION 


Background 

Although  there  are  some  notable  exceptions,  there  is  a recognizable  trend 
upwards  in  the  cost  and  complexity  of  most  new  military  hardware.  Each  ex- 
pensive new  system  that  fails  in  the  field  to  meet  its  expected  standards  of 
performance  (or  is  found  by  operational  tests  to  have  significant  use  problems) 
testifies  to  deficiencies  in  the  techniques  of  engineering  management  under 
which  it  was  produced.  Certain  recent  programs  have  shown  that  early  design 
mistakes  — particularly  those  which  affect  human  performance  requirements 
within  the  system  — can  remain  undetected  until  quite  late  in  development. 
Decreasing  amounts  of  research  and  development  funds  available  for  each 
project,  plus  increasing  public  interest  in  the  Defense  Department's  materiel 
acquisition  process,  have  intensified  the  search  for  management  techniques 
whose  employment  will  yield  an  improvement  in  system  performance  (38,  p.l). 

In  the  late  1960's,  it  came  to  the  attention  of  certain  Defense  Depart- 
ment managers  that  the  unsatisfactory  performance  record  of  certain  systems 
somehow  involved  the  personnel  who  had  to  operate  and  maintain  them.  An 
increased  emphasis  on  "human  factors"  was  directed  for  new  systems  under 
development.  This  emphasis  produced  some  new  regulations  and  policies  and 
several  new  contract  data  items.  One  of  these.  Data  Item  1334  (DI-H-1334), 
entitled  "Report(s)  of  Human  Factors  Engineering  Test,"  was  initially  intended 
to  support  then -Secretary  of  Defense  Laird's  policy  of  "fly  before  you  buy"  by 
identifying  early  in  the  development  cycle  potential  man-machine  interface 
problems  which  might  ultimately  reduce  system  effectiveness  or  reliability. 

Historically,  the  discipline  of  human  factors  engineering  (HFE)  has  been 
handicapped  in  identifying  such  problems  because  there  was  not  enough  access 
to  drawings  and  models  during  the  early  design  stages.  This  lack  of  access, 
often  founded  in  cost  and  schedule  constraints,  has  increasingly  come  to  be 
viewed  as  counterproductive.  The  Arniy  spends  far  more  money  in  product 
improvements,  retrofits,  upgrading  personnel  aptitude  requirements,  and 
lengthening  training  than  it  would  have  spent  conducting  human  engineering 
tests  and  requiring  design  changes  during  original  prototype  development. 
Consequently,  DI-H-1334's  purpose,  in  both  exploratory  and  advanced  develop- 
ment contracts,  is  to  identify  problems  sufficiently  early  to  permit  their 
solution  by  contractor's  designers  as  an  integral  part  of  early  development. 

Following  the  introduction  of  DI-H-1334,  several  revisions  were 
identified  which  would  more  clearly  define  the  task  performances  to  be 
observed  and  clarify  the  manner  of  reporting  the  human  performance  data. 
Accordingly,  DI-H-1334A  has  been  issued  to  specify  the  manner  in  which  human 
factors  engineering  testing  shall  be  conducted  and  reported. 
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Basically,  the  test  methodology  contained  in  DI-H-1334A  works  by 
identifying  four  factors  (the  man,  his  training,  what  he  has  to  do,  and  the 
configuration  of  the  equipment  on  which  he  works)  and  then  assessing  their 
ccnpatibility . In  addition  to  that  assessment,  data  are  also  provided  that 
can  be  used  to  (1)  verify  that  the  human  performance  tasks  required  in  the 
system  can,  in  fact,  be  performed  by  humans,  (2)  accurately  identify  the 
aptitudes  and  skills  required  by  system  personnel,  (3)  establish  the  adequacy 
of  the  proposed  training  program,  and  (4)  confirm  that  the  materiel  itself  is 
adequately  human-engineered.  The  requirements  of  DI-H-1334A  are  accomplished 
by  analyzing  operator  performance  requirements,  followed  by  the  acquisition 
of  performance  data,  along  with  observations  of  potentially  adverse  factors 
such  as  human  errors,  equipment  incompatibilities,  intereference  by  other 
operators,  and  safety  hazards. 

Since  1972,  when  DI-H-1334A  was  approved,  a series  of  events  has 
considerably  expanded  the  potential  usefulness  of  the  data  item: 

1.  System  Reliability  Predictions:  In  the  search  for  specific  manage- 
ment techniques  which,  when  employed,  are  likely  to  insure  greater  system 
effectiveness,  the  so-called  "system  reliability  program"  has  received 
prominent  attention  from  military  managers.  Most  of  these  programs  attempt 
to  calculate  a decimal  number  representing  the  probability  that  a given 
system  (component)  will  do  what  it  is  supposed  to  do  for  a stated  period  of 
time  in  a given  environment.  These  probabilities,  in  turn,  are  theoretically 
sensitive  to  design  changes  and  can  thus  provide  useful  information,  both  in 
the  evaluation  of  competing  concepts  and  in  the  assessment  of  an  evolving 
concept.  Certain  such  reliability  estimates  have  been  surprisingly  correct; 
others  have  not,  and  their  acceptance  has  resulted  in  serious  engineering 
errors.  When  reliability  predictions  fail,  it  is  often  not  so  much  a function 
of  system  complexity  as  the  amount  of  human  performance  involved.  A reli- 
ability program  which  ignores  the  human  component  has  a high  likelihood  of 
providing  the  potential  user  with  an  inflated  prediction  of  system  effective- 
ness, particularly  where  a "system"  can  perform  its  function(s)  only  if  the 
operator(s)  perform  critical  tasks  accurately  and  promptly.  Recognition  of 
this  fact  is  today  far  more  widespread  than  a decade  ago,  when  an  Aerospace 
Industries  Association  publication  dared  to  claim,  "...equipment  reliability 
in  and  of  itself,  without  considering  the  personnel  aspect,  is  only  50%  of 
the  reliability  picture"  (32).  In  order  to  make  system  reliability 
predictions  more  accurate,  data  are  required  to  show  the  performance 
reliability  of  the  human  operators  executing  the  tasks  assigned  to  them  in 

the  system.  A modest  program  was  undertaken  by  the  U.S.  Army  Human  Engineering 
Laboratory  in  1972  to  show  how  such  data  could  be  gathered  and  integrated  into 
complex  mathematical  system  reliability  models.  One  of  the  findings  of  th  s 
program  was  that  DI-H-1334A  could  be  used  to  collect  the  needed  data  (37). 

2.  Logistics  Support  and  Analysis:  As  some  Army  programs  strived  to 
increase  the  quality  of  performance  of  systems  under  development,  other 
programs  were  trying  to  reduce  the  maintenance  burden  of  such  new  systems. 

In  particular,  the  Integrated  Logistics  Support  (ILS)  program  and,  more 
recently,  the  Logistics  Support  Analysis  Record  (LSAR)  were  instituted  to 
assure  that  maintenance  concerns  would  be  considered  adequately  during  system 
design  and  evaluation.  The  intent  of  these  programs  is  to  identify  accurately 
the  personnel  skills  which  would  be  needed  within  the  maintenance  organizations 
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supporting  the  new  system,  to  minimize  the  personnel  required  to  perform  the 
maintenance,  and  to  insure  that  those  personnel  with  the  necessary  skills 
would,  in  fact,  be  available  in  the  maintenance  support  units.  None  of  these 
determinations  can  be  made  accurately  without  knowledge  of  what  maintenance 
tasks  are  to  be  performed  and  how  well  the  maintenance  personnel  can  be 
expected  to  perform  those  tasks.  Data  Item  DI-H-1334A  is  ideally  suited  as  a 
mechanism  to  identify  and  assess  these  maintenance  tasks  during  the  early 
stages  of  system  design.  Indeed,  the  data  generated  under  the  guidance  of 
DI-H-1334A  can  provide  the  HFE  data  needed  in  the  Logistics  Support  Analysis 
Record. 


3.  Training:  Within  the  environment  of  a voluntee''  Army,  it  is 
extremely  important  to  minimize  the  amount  of  time  an  individual  spends  in 
operator  or  maintainer  training,  and  to  maximize  the  amount  of  time  that 
individual  spends  in  the  unit  actually  operating  and  maintaining  equipment. 

In  addition,  the  weapon  systems  of  the  modern  Army  are  becoming  increasingly 
sophisticated,  requiring  highly  skilled  operators.  These  two  considerations 
place  a heavy  responsibility  upon  system  proponents  and  developers  to  assess 
adequately  and  early  the  training  requirements  imposed  by  a new  system. 

Recent  program  introductions,  includinq  the  Skill  Performance 

Aids  (SPA)*  <ind  the  Cost  and  Training  Effectiveness  Analysis  (CTEA) 
clearly  point  to  the  recognized  need  to  assess  the  effectiveness  and  adequacy 
of  training  early  in  system  development.  Both  programs  stress  the  concepts 
of  (1)  analyzing  the  total  weapon  system  and  the  training  subsystem,  (2) 
emphasizing  quantitative  assessment  of  measures  of  effectiveness  that  are 
meaningful  to  decision  makers,  and  (3)  performance-oriented  training.  The 
key  element  of  both  programs  is  assessment  of  operator  and  maintainer 
performance  which  has  been  identified  through  an  adequate  task  analysis. 
Additionally,  the  U.S.  Army  Human  Engineering  Laboratory  has  recently  con- 
cluded the  first  phase  of  a program,  entitled  "Low-Cost  Ownership"  (46),  to 
examine  a new  training  concept  which  appears  to  have  a significant  potential 
for  reducing  formal  training  time  without  adversely  affecting  the  quality  of 
the  individual's  performance.  At  the  core  of  this  new  concept  is  an  early 
knowledge  of  the  detailed  human  performance  requirements  for  operations  and 
maintenance.  Data  Item  DI-H-1334A  specifically  requires  (1)  that  the  tasks 
for  each  individual  operating  or  maintaining  the  system  be  identified,  (2) 
that  the  performance  of  these  tasks  be  measured  quantitatively,  and  (3) 
that  the  test  participant's  training  be  assessed.  Thus,  DI-H-1334A  can 
provide  much  of  the  information  needed  by  spa  and  CTEA  to  assess  training 
effectiveness. 

4.  Test  and  Evaluation:  In  the  process  of  implementing  the  Life  Cycle 
System  Management  Model  (57),  the  Army  has  promulgated  several  regulations 
specifying  the  manner  in  which  test  and  evaluation  shall  be  conducted.  The 
new  Army  regulation  on  testing,  AR  70-10,  encourages  the  simultaneous  conduct 
of  developmental  tests  (DT)  and  operational  tests  (OT),  and  makes  it  clear 
that  contractor  facilities  may  be  used  for  both  developmental  and  operational 
tests  (76).  Arrny  pamphlet  602-1  (62),  as  well  as  Army  regulation  AR  70-10, 
clearly  authorizes  the  U.S.  Army  Materiel  Development  and  Readiness  Command 
(DARCOM)  to  measure  system  performance  in  DT  with  soldiers,  thus  testing  the 
system  under  conditions  similar  to  OT  (i.e.,  in  the  hands  of  user  troops). 
Another  recent  Army  policy.  Single  Integrated  Development  Test  Cycle  Policy 

*formerly  "Integrated  Technical  Documentation  and  Training  (ITDT) 
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(SIDTC),  states  that  developmental  testing  shall  be  structured  as  an 
integrated  test  cycle  to  assure  that  the  contractor,  developer,  tester,  and 
evaluator  interact  to  minimize  test  cost  and  to  maximize  the  use  of  test 
data  (56).  The  intent  of  these  regulations  and  policies  is  to  integrate  as 
much  testing  as  possible  and  to  avoid  duplication  of  tests.  At  least  as  far 
as  developmental  testing  is  concerned,  this  intent  also  extends  to  using 
contractor-generated  data,  where  possible.  Inherent  in  these  regulations  and 
policies  is  the  notion  that  successful  development  decision  making  lies  in 
independent  evaluation  of  valid  data,  rather  than  the  independent  conduct  of 
tests. 

In  keeping  with  the  intent  of  the  new  regulations,  the  Artny  agencies 
responsible  for  test  and  evaluation  are  specifying  the  procedures  to  be  used 
in  evaluating  the  human  factors  engineering  (HFE)  of  a system  in  development. 
The  Test  and  Evaluation  Command  (TECOM)  has  published  the  Human  Factors 
Engineering  Data  Guide  for  Evaluation  (HEDGE)  (52)  and  the  Operational  Test  and 
the  Evaluation  Agency  (OTEA)  is  sponsoring  the  development  of  the  Human 
Resources  Test  and  Evaluation  System  (HRTES)  (17).  In  both  cases,  the  test 
and  evaluation  procedures  rest  on  identifying  the  activities  to  be  performed 
by  the  operators  and  maintainers.  Data  Item  DI-H-1334A  provides  a mechanism 
for  a contractor  to  obtain  and  report  human  performance  data  in  a form  which 
can  be  used  by  the  developmental  tester  and  the  operational  tester  to  evaluate 
the  impact  of  operator  and  maintainer  performance  on  system  performance. 


Purpose 

It  has  been  the  experience  of  the  human  factors  community,  on  both  sides 
of  the  procurement  fence,  that  requirements  for  human  factors  analyses,  tests, 
and  data  are  among  the  first  to  be  waived  or  neglected  in  actual  development 
contracts.  At  the  root  of  this  problem  is  a lack  of  understanding,  on  the 
part  of  many  engineers,  managers,  and  monitors,  of  both  the  purposes  and 
procedures  associated  with  HFE  data.  They  may  know  what  data  are  called  for, 
but  they  do  not  know  how  to  obtain  these  data  and,  even  more  important,  they 
do  not  know  how  the  data  should  be  used.  The  overall  purpose  of  this  report 
is  to  provide  system  engineers,  managers,  and  monitors  with  descriptions  of 
how  to  plan  and  administer  HFE  testing  and  how  to  collect,  analyze  and 
interpret  HFE  test  data. 

The  specific  objectives  of  this  report  are  to;  (1)  describe  how  to 
conduct  and  report  an  HFE  test  according  to  the  requirements  of  DI-H-1334A, 

(2)  detail  the  expenditures  in  time  and  money  associated  with  the  conduct  of 
an  HFE  test,  (3)  provide  examples  of  HFE  test  reports  for  systems  in 
"experimental  prototype"  and  "advanced  development"  phases  of  development, 

(4)  describe  the  uses  of  the  obtained  HFE  test  data  as  a function  of  system 
development,  and  (5)  explain  the  impact  of  the  D1-H-1334A  findings  on  a 
materiel  development  program. 

This  document  is  written  for  government  contract  monitors,  contract 
project  directors,  and  contractor  HFE  personnel.  The  guidelines  for  con- 
ducting and  reporting  on  the  HFE  test  are  intended  for  experienced  HFE 
personnel  (degree  in  experimental  psychology  or  human  factors,  or  background 
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in  human  factors,  statistics,  and  test  and  evaluation).  Therefore,  detailed 
descriptions  of  the  recoirmended  resting  techniques  are  not  provided.  Instead, 
techniques  are  discussed  briefly'and  extensive  references  are  made  to  the 
literature  on  human  factors  test  and  evaluation  methodology. 

Questions  of  what  data  are  to  be  collected,  how  they  are  to  be  collected, 
and  how  the  data  can  be  used  are  discussed  in  Chapters  2,  3,  and  4 of  this 
report.  Chapter  2 is  a guide  for  planning,  conducting,  analyzing,  and 
reporting  on  an  HFE  evaluation  according  to  the  requirements  of  DI-H-1334A. 
Procedures  for  managing  the  HFE  evaluation,  allocating  test  personnel, 
developing  test  cost  estimates,  etc.,  are  also  contained  in  Chapter  2.  This 
information  will  aid  project  managers  in  the  administration  and  organization 
of  an  HFE  evaluation.  Chapter  2 will  also  help  the  contract  monitor  supervise 
the  HFE  evaluation.  The  explanations  of  the  DI-H-1334A  requirements  will 
assist  the  contract  monitor  to  understand  and  monitor  HFE  tests  and  insure 
that  all  of  the  requirements  of  DI-H-1334A  are  satisfied. 

Chapters  3 and  4 supplement  the  guidelines  given  in  Chapter  2 by  giving 
detailed  examples  of  HFE  reports,  written  according  to  the  requirements  of 
DI-H-1334A.  Chapter  3 presents  the  HFE  test  report  of  a system  in  the 
experimental  prototype  stage  of  development.  This  sample  HFE  report  focuses 
on  determining  the  feasibility  of  human  performance,  the  appropriateness  of 
the  tasks  allocated  to  the  operator  and  to  the  machine,  and  the  adequacy  of 
the  workspace  layout.  This  report  also  demonstrates  procedures  for  conducting 
HFE  tests  on  mock-up  equipment. 

Chapter  4 contains  the  HFE  test  report  of  an  advanced  development 
prototype.  The  emphasis  in  this  stage  of  development  is  on  determining  the 
capability  of  the  operator  to  perform  his  assigned  tasks  within  his  prescribed 
time  and  error  standards.  This  test  report  also  evaluates  the  adequacy  of 
operator  selection  and  training,  as  well  as  the  equipment  configuration. 

Chapter  5 discusses  implications  of  human  performance  tests.  The  uses 
to  which  data  can  be  applied  and  the  problem  associated  with  conducting  HFE 
tests  are  also  described.  By  describing  how  HFE  test  data  can  be  applied 
and  the  benefits  of  collecting  the  data,  program  managers  can  better 
appreciate  the  need  for  HFE  testing. 


Approach 

Sample  Materiel  Development  Programs.  Two  HFE  tests  were  conducted  to 
provide  samples  of  procedures  for  planning,  conducting,  analyzing,  and 
interpreting  HFE  test  results  at  different  stages  of  system  development.  The 
sample  materiel  development  programs  were  (1)  the  mission  control  station  of 
Compass  COPE,  and  (2)  the  Communication  Control  Unit  (CCU)  of  TACFIRE. 

Compass  COPE  is  a system  of  multiple,  remotely-piloted  vehicles  (RPV's) 
with  a mission  of  high  altitude,  long  duration,  long  range  data  acquisition. 
Up  to  five  vehicles  are  to  be  remotely  controlled  simultaneously  from  a 
single  mission  control  station.  Compasss  COPE  is  in  the  "experimental 
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prototype"  stage  of  development.  For  the  purpose  of  the  HFE  test,  a static 
mock-up  of  the  pilot's  and  copilot's  console  was  constructed  (Figure  1). 

The  ecu  of  TACFIRE  is  designed  to  control  digital  and  voice  communication 
traffic  associated  with  a computerized  military  operations  center.  Figure  2 
shows  the  CCU  installed  in  the  TACFIRE  Fire  Direction  Center  Shelter. 

Although  the  CCU  is  currently  in  the  Low  Rate  Initial  Production  phase  of 
materiel  development,  for  the  purposes  of  this  report,  it  was  assumed  to  have 
just  completed  advanced  development  (57). 

Sample  Program  Selection  Criteria.  Modern  materiel  systems  depend  to 
an  ever-increasing  extent  on  built-in  data  processing  capability.  Systems 
such  as  TACFIRE,  TOS,  REMBASS,  and  others  already  fall  into  this  class;  many 
systems  for  vehicle  and  weapons  guidance,  communications,  intelligence,  and 
equipment  maintenance  are  being  converted  to  a computer-based  configuration. 

In  such  systems,  the  operator's  role  is  primarily  that  of  an  information 
processor  and  system  effectiveness  depends  on  his  ability  to  obtain, 
understand,  and  transmit  information.  In  addition,  system  operators  must 
frequently  interact  with  software  as  well  as  hardware  components.  The 
development  projects  evaluated  in  this  report  are  representative  of  these 
modern  materiel  systems. 


Figure  1.  Static  mock-up  of  the  Ground  Control  Station  (GCS) 
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GUIDE  TO  DATA  ITEM  DI-H-1334A 


The  checklist  shown  in  Table  1 is  a guide  for  performing  an  HFE  test 
according  to  the  requirements  imposed  by  DI-H-1334A.  This  checklist 
contains  all  the  activities  required  by  DI-H-1334A;  thus,  the  checklist 
serves  as  a handy  reference  for  developing  HFE  test  plans,  conducting  tests 
and  preparing  HFE  test  reports.  Data  item  DI-H-1334A  is  shown  in  Table  2. 
Block  10  of  the  data  item  lists  specific  requirements  for  the  HFE  tests 
and  reports. 

Each  section  of  this  chapter  is  labeled  in  accordance  with  Table  1 
and  gives  detailed  explanations  of  the  activities  specified  in  the  test 
checklist.  Frequent  references  are  made  to  the  two  test  reports  contained 
in  chapters  3 and  4.  These  two  sample  reports  illustrate  methods  for 
accomplishing  the  required  test  activities  for  projects  in  the  experimental 
prototype  and  advanced  development  prototype  phases  of  materiel  development 
Throughout  the  discussions  of  the  HFE  test  activities,  extensive  references 
to  the  human  factors  literature  are  made  to  provide  further  descriptions 
of  test  procedures. 


Test  Administration 

While  methods  for  managing  hunan  factors  tests  vary  as  a function  of 
the  contractor's  organization,  a single  person  should  normally  be  appointed 
to  manage  all  of  the  activities  described  in  this  chapter.  A useful  guide 
for  managing  projects  is  provided  by  Abramson  and  Kennedy  (1). 

The  first  task  of  the  test  manager  will  be  to  review  the  test 
requirements  described  in  this  chapter  and  block  10  of  the  data  item. 

With  a comprehensive  view  of  the  test  requirements,  the  manager  can  develop 
the  preliminary  test  milestones,  specify  the  personnel  requirements,  and 
prepare  the  test  budget. 

Milestone  Development.  The  test  manager  should  develop  test 
milestones  to  provide  a guideline  for  timely  execution  of  test  activities 
and  for  scheduling  test  personnel,  equipment,  and  facilities.  A milestone 
is  an  activity  with  a definable  end  point  or  product.  Test  activities  of 
the  HFE  checklist  shown  in  Table  1 identify  the  test  milestones  to  be 
completed  in  a HFE  test.  Table  3 shows  the  milestone  chart  for  the  HFE 
test  of  the  TACFIRE  CCU  (Sample  of  Advanced  Development  Prototype).  As 
shown  in  Table  3,  test  administration  activities  continue  throughout  the 
test  program. 

Manpower  Specification.  Manning  requirements  for  a D1-H-1334A 
test  depend  upon  the  number  of  personnel  positions  in  the  system  and  the 
complexity  of  the  tasks  being  evaluated.  The  greater  the  number  of  system 
personnel  and  the  more  complex  their  performance  requirements,  the  greater 
the  amount  of  manpower  required  to  conduct  the  test.  A test  manager  is 
required  to  oversee  the  tests  and  to  insure  that  all  required  data  are 
properly  collected.  In  addition  to  the  test  manager,  one  test  person  is 
generally  required  for  each  personnel  position  being  evaluated,  depending 


13 


TABLE  1 


HFE  Test  Checklist 


TEST  ACTIVITIES  SOURCE  OF  INFORMATION 


1.  TEST  ADMINISTRATION 

1.1  Milestone  Development  1334A,  HEL  TM  29-76,  System  test  milestones 

1.2  Manpower  Specification  Available  organization  personnel 

1.3  Budget  Preparation  Manpower  needs.  Organization  budget  procedures 

2.  TASK  GROUP  DESCRIPTION 

2.1  Task  Group  Identification  MOS  or  Analysis  of  personnel  position 

2.2  Task  Analysis  Equipment  configuration.  Task  allocation 

2.3  Performance  Standards  Identification  System  performance  specifications 


3.  TEST 

PLANNING  AND  DESIGN 

3.1 

Test  Objectives  Specification 

1334A,  Previous  HFE  tests.  System  contract.  Stage 
of  development 

3.2 

Test  Equipment  Design  and  Selection 

System  description,  TASA,  Stage  of  development 

3.3 

Test  Environment  Measurement 

Analysis  of  operational  environment.  Stage  of 
development 

3.4 

Test  Participant  Selection 

Task  group  Identification,  Expected  test  variability, 

Stage  of  development 

3.5 

Test  Participant  Training 

Participants'  experience.  System  development  stage 

3.6 

Data  Acquisition  and  Analysis  Planning 

System  development  stage.  Statistics  literature 

3.7 

Test  Segment  Development 

TASA,  Stage  of  development.  Preliminary  hazard  analysis 

3.8 

Test  Schedule  Preparation 

Test  segments.  Number  of  test  personnel 

3.9 

Test  Dry  Run  Execution 

Test  plan.  Test  segments 

4.  HFE  TEST  EXECUTION  DATA  ACQUISITION 

4.1 

Pretest  Procedures 

Prepared  questionnaires.  Training  materials 

4.2 

Test  Execution 

Objective  performance  measures.  Test  plan 

5.  DATA  ANALYSIS 

5.1 

Objective  and  Subjective  Data 
Sunmarlzatlon 

Performance  measures 

5.2 

Problem  and  Error  Description 

Data  records 

5.3 

Man-Machine  Incompatibilities 

Data  records.  Observations  of  test  oersonnel 

5.4 

Human  Performance  Impact  Assessment 

Performance  standards  or  baseline  task  as  comparison 

6.  REPORT  PREPARATION 

1334A,  Test  plan.  Analyzed  data 
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TABLE  2 

Data  Item  DI-H-1334A 


DATA  ITEM  OESCRIPTIOM 


IDINTiriCATlOH  NOItl 


R«port(a)  of  Human  Factors  Engineering  Test 


Any  I D1-B-133AA 

nwumrtiTr 


>■  eSKMlVTION/PUNPOaS 


Used  to  detenlne  whether  end  to  what  level  or  standard (e) 
each  trained  individual  can  perfon  in  the  specified  eequence 
all  of  the  perforaence  taeks  aesigned  to  hla  in  a syeteei;^  to 
detenlne  whether  and  to  what  extent  hie  perfonence  is  affect- 
ed by  equlpaent  configuration,  the  perfononce  of  other  syeten 
pereonnel,  or  both;  end  to  aeaess  the  lapect  of  the  meaeured 
huun  performance  on  the  ettelnaent  of  eystem  performance 
goals. 


1 October  1976 


DBmB-pr 


•.  DOC  peeuiNao 


APPneVAC  LIMITATIOM 


>.  ADALtcAneM/iNTeeDacArioMaMip 


Servee  ae  the  principal  aoene  of  eubstantletlng  Che 
feasibility  of  required  human  performance,  the  accuracy  of 
the  personnel  aalecclon  criteria,  the  adequacy  of  the  training 
program,  and  the  acceptability  of  the  man-uchlne  Interfaces. 
Whan  DI-H-1313  (HFC  Teat  Flan)  la  also  s contractual 
requlrasMnt,  It  shall  Include  a section  which  describes  In 
detail  the  efforts  necessary  Co  accomplish  tha  test  described 
below. 


This  DID  supersedes  DI-H-1334. 


fiKffjr 

AR  70-10 

AR  602-1 

MIL-H-A68SS 

MIL-STD-1A72 

MIL-STD-147A 

MIL-HI>BK-759 

REL  TM  29-76 


maiiiiim  AA  •UttltT 


MCAL  NUMseme 


The  Raporc(s)  of  HFE  Test  shall  be  submitted  by  the  contractor  to  the  procuring  activity 
for  each  personnel  position  In  tha  eystem  being  developed.^  All  of  the  operations  and 
aulntenance  casks  required  of  the  soldier  assignsd  to  a personnel  position  are  rafarred 
to  as  the  "task  group"  of  that  position.  This  date  Item  nay  be  submitted  Incrementally 
Co  facilitate  use  of  test  results  (l.s.,  portions  of  a task  group,  or  only  one  cask 
group,  nay  be  tested  and  reported  separately).  However,  this  data  Item  shall  not  be 
considered  coapleca  and  sceepcable  until  all  task  groups  In  tha  system  end  tha  Intsr- 
actlona  of  each  have  been  taatad  and  reported.  The  Raporc(s)  of  HFS  Test  shall  Include, 
but  not  necesaarlly  be  limited  to,  sections  described  by  the  following  paragraphs  and 
notes: 


1.  INTRODUCTION 

a.  Identification  of  task  groups  (or  portions  thereof)  reported. 


b.  For  each  task  group  or  portion  thereof  reported,  a brief  summary  of  major  tasks 
(Including  Inceraccloas  with  othar  operators  and  aalntalnars)  and  a brief  description  of 
the  physical  ares  la  which  they  are  to  ba  perfomed  whan  the  eystM  Is  fielded. 


c.  Data(s),  locatlon(e)  and  naaas(s)  of  Indlvldual(e)  present  and  supervising 
conduct  of  the  HFE  coat. 


2.  TEST  FREFAIATION 


Stacemenc  of  the  task  groups  (or  portions  thsreof)  being  reported,  (a  list  in 
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2.  TEST  PREPARATION  (continued) 

sequential  order,  of  all  the  discrete  perfomance  tasks  which  will  be  required 
of  each  person  In  Che  systesi. 

b.  Stateaent  of  (or  reference  to)  any  huaan  performance  standards  (e.g., 

".9  probability  of  gunner  launching  missile  within  10  seconds  after  detecting 
target")  or  assumed  contribution  to  error  (e.g.,  "aiming  error  less  than  3 mils") 
contained  In  system  development  documents.  If  none,  so  state. 

c.  Description  of  envlroraaent  at  each  distinct  location  of  human  performance. 
(Include  noise  and  Illumination  levels,  shock  and  vibration,  air  temperature 

and  humidity,  and  ventilation.  Also  state  the  concentration  and  exposure  tine 
of  test  participants  to  any  toxic  or  hazardous  substances;  and  state  whether 
that  exposure  was  or  was  not  within  the  applicable  safety  limits  for  each 
substance.) 

d.  Description  of  test  participants.^  For  each  participant,  state  age, 
weight,  body  dimensions  relevant  to  performance  tasks  (see  paragraphs  3.1  and 
5.6,  MIL-STD-1A72) , visual  acuity  and  hearing  levels,  any  known  physical  dis- 
abilities, and  score  from  a standardized  measure  of  general  Intelligence. 

e.  Description  of  Individual  clothing  and  equipment  (Including  all  clothing 
and  equipment  worn,  carried  or  otherwise  borne  on  the  body  such  as  weapon, 
communications  equipment,  headgear  and  protective  mask^) . 

f.  Type  and  amount  (In  hours)  of  system-specific  pre-test  training 
(differentiating  "hands  on"  practice  from  other  training)  given  to  test 
participants;  and  type,  content  and  results  of  training  assessment^  used.  Also 
state  time  Intervals  between  end  of  training,  training  assessment,  and  start 

of  HFE  tests  being  reported. 

g.  Description  of  mockup  or  equipment  on  which  HFE  test  Is  conducted 
(Including  material  used  and  type  of  fabrication;  dimensions;  and  cross- 
reference  to  blueprints,  drawings  or  sketches). 

h.  Identification  of  devlatlon(s)  during  the  HFE  test  from  conditions  of 
expected  use  (paragraph  l.b.  above);  narrative  explanation  of  rea8on(s)  for 
devlation(s) , and  presumed  effect  of  such  devlatlon(s)  on  the  validity  of 
generalizations  from  test  data. 

3.  DATA  COLLECTION  TECHNIQUES 

a.  Idantlf icatlon  of  the  quantitative  and  qualitative  measures  of  both  human 
and  system  performance. 

b.  Description  of  methods,  procedures  and  Instrumentation  used  In  data 
collection. 

c.  Description  of  techniques  used  for  data  reduction,  statistical  techniques 
employed,,  and  confidence  levels  selected. 


Page  2 of  5 Pages 
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4.  HFE  TEST  RESULTS 

a.  Summaries  of  quantitative  human  and  system  performance  data. 

b.  Summaries  of  qualitative  data  (Including  questionnaires.  Interviews, 
checklists,  etc.). 

5.  DESCRIPTION  OF  HUMAN  PERFORMANCE  ERRORS 

a.  Narrative  description,  with  photograph(8)  if  appropriate,  of  each  error. 
Include  frequency  of  occurrence  of  each  error  during  test. 

b.  Consequence  (brief  statement  of  the  immediate  effect  of  the  error  on 
system  operation). 

c.  Causes  (isolation  of  the  Innedlate  cause  of  each  actual  performance 
error  and  Identification  of  the  events,  conditions,  operator  workload, 
environmental  factors  and  equipment  configurations  which  may  have  contributed 
to  it). 


d.  Explanation  of  participants  making  errors  and  the  reasons  for  the 
errors . 


e.  Recommended  solutions  (stated  in  terms  of  equipment  redesign,  alteration 
of  tasks,  personnel  selection  and/or  training).  Provide  rationale. 

6.  DESCRIPTION  OF  INCOMPATIBILITIES  AMONG  HUMAN  PERFORMANCE  AND  EQUIPMENT 

a.  Identification 

(1)  During  the  test  what  tasks  of  one  task  group  Interfered  with  the 
performance  of  which  tasks  of  another  task  group?  If  none,  so  state. 

(2)  During  the  test  what  two  or  more  items  of  equipment  were  found 

to  be  incompatible  and  yhat  human  performance  was  affected?  If  none,  so  state. 

(3)  During  the  test  what  human  performance  was  adversely  affected  by 
what  equipment  configurations  or  characteristics?  (Identify  controls  and/or 
displays  needed  but  not  present.)  If  none,  so  state. 

b.  Recommended  solutions  (stated  in  terms  of  equipment  redesign,  alteration 
of  tasks,  personnel  selection  and/or  training).  Provide  rationale. 

7,  DESCRIPTION  OF  OBSERVED  SAFETY  HAZARDS 

a.  Narrative  description,  with  photograph(s)  if  appropriate,  of  each  safety 
hazard  identified  during  the  test. 

b.  Frequency  each  hazard  was  encountered  by  test  participants. 

c.  Severity  and  consequence  of  each  hazard. 

Page  3 of  5 Pages 
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7.  DESCRIPTION  OF  OBSERVED  SAFETY  HAZARDS  (continued) 

d.  Recomnended  action  to  eliminate  or  minimize  hazard  (stated  In  terms  of 
equipment  redesign,  alteration  of  tasks,  personnel  selection  and/or  training). 
Provide  rationale. 

8.  ANALYSIS  OF  LMPACT  OF  HUMAN  PERFORMANCE  ON  ATTAINMENT  OF  SYSTEM  PERFORMANCE 
GOALS 

a.  Statement  of  (or  reference  to)  system  performance  goals. 

b.  Narrative  explanation  of  reasons  why  any  human  performance  tasks  required 
by  present  equipment  design  are  not  feasible;  or  why  any  standards  presently  set 
for  spe. iflc  human  performance  tasks  are  unattainable.  (If  all  human  performance 
requirements  are  feasible  and  any  standards  set  appear  to  have  been  met,  so  state.) 

c.  Narrative  explanation  of  how  measured  human  performance  errors  In 
operations  and  maintenance  can  affect  reliability  and  availability. 

d.  Narrative  explanation  of  how  measured  human  performance  times  and  error 
frequencies  and  magnitudes  can  affect  system  effectiveness. 

e.  Narrative  explanation  of  how  system  performance  goals  would  be  affected 
by  Implementing  the  solutions  recoimnended  In  subparagraphs  S.e.,  6.b.  and  7.d 
(above) . 

9.  CONCLUSIONS 

a.  Summary  of  major  test  findings. 

b.  Sumnary  of  Implications  for  system  performance  (reliability,  availability 
and  effectiveness)  of  the  human  performance  measured  In  this  test. 

c.  List  of  recoimnended  changes  to  equipment  configuration,  human  performance 
tasks,  personnel  selection  and/or  training  (In  order  of  decreasing  Importance). 


NOTES 

1.  "System"  Is  defined  as  a composite  of  hardware  (and  any  software),  the  personnel 
who  operate  and  maintain  It,  the  training  they  receive,  and  the  tools  and  reference 
publications  they  use. 

2.  A task  group  Is  "In  the  system"  If  a soldier  must  perform  tasks  on  equipment 
produced  by  the  contractor  or  a subcontractor  (e.g.,  a squad  leader  viould  not  be 
"In"  a new  machlnegun  system  even  though  he  supervised  the  mschlnegunner ; but  the 
assistant  gunner  and  the  unit  maintenance  technician  would  be). 

3.  Performance  data  from  at  least  three  different  individuals  are  required  for 
each  cask  tested. 
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DI-H-1334A  (Cont'd) 


NOTES  (Continued) 

4.  Unless  otherwise  stated  in  Block  16  of  that  sequence  on  DD  Form  3423  which 
makes  this  data  Item  a contractual  requirement,  performance  data  (reported  In 
paragraph  4 above)  shall  be  gathered  while  test  participants  carry  the 
protective  mask  In  an  approved  manner  on  the  body;  but  do  not  wear  It. 

5.  Each  test  participant  shall  be  tested  to  determine  his  or  her  comprehension 

of  task  group  performance  requirements  prior  to  the  gathering  of  the  data  reported 
In  paragraph  4 above.  The  purpose  of  this  test  Is  limited  to  verifying  that  each 
test  participant  knew  what  he  or  she  was  supposed  to  do  with  the  equipment 
before  data  gathering  began;  this  requirement  is  not  primarily  Intended  to  be  an 
assessment  of  the  New  Equipment  Training  (NET)  Program. 


AUJ.  OOVBSNIRHT  niNTmO  OmCI:  l»T7-701-UI/tT«T 
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TABLE  3 


Sample  Test  Milestone  Chart 
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upon  the  method  of  data  collection.  This  test  person  is  responsit  e for 
monitoring  test  participant  performance  and  for  recording  test  data. 

To  provide  a yardstick  for  estimating  test  manpower  requirements,  the 
number  of  man-hours  required  to  complete  the  two  sample  HFE  tests  is  shown 
in  Table  4.  A substantial  proportion  of  the  time  spent  in  conducting  an  HFE 
test  will  be  spent  in  planning.  Efficient  planning  will  facilitate  data 
acquisition  and  analysis  and  will  contribute  to  cost  reduction.  Test 
planning  was  a major  effort  of  the  sample  tests  and  consumed  about  thirty- 
four  percent  of  the  total  man-hours  expended.  Additional  man-hours  may  be 
required  for  equipment  construction.  In  the  HFE  test  of  the  RPV  control 
station,  one  hundred  thirty  man-hours  were  required  for  mock-up  design  and 
construction. 

Budget  Preparation.  Specific  budget  requirements  for  the  human  factors 
test  will  depend  primarily  on  the  contractor's  in-house  budget  procedures. 

The  manpower  requirements  for  the  sample  HFE  tests  provide  a basis  for 
estimating  test  costs.  Additional  factors  which  may  influence  the  budget 
estimates  include  the  amount  of  travel  to  and  from  the  test  site,  any  payment 
for  test  participants'  time,  and  any  equipment  required  for  a mock-up  and 
for  conducting  the  test.  To  provide  guidance  in  preparing  test  budgets, 
the  costs  for  the  sample  HFE  tests  are  shown  in  Table  5. 


Task  Group  Description 

Task  Group  Identification.  The  objective  of  the  HFE  test  is  to  evaluate 
the  performance  of  equipment  operators.  To  perform  the  evaluation,  the 
operator's  task  group  must  be  defined.  As  defined  by  DI-H-1334A,  a task 
group  consists  of  all  operations  and  maintenance  tasks  assigned  to  a single 
personnel  position.  The  design  of  the  machine  and  its  intended  use  determines 
what  the  operator  must  do.  For  this  reason,  the  general  equipment  configura- 
tion must  be  specified  before  the  task  analysis  is  completed.  In  the  actual 
design  process,  the  task  analysis  may  identify  needed  design  revisions. 

Thus,  system  design  and  task  analysis  is  an  iterative  process.  The 
configuration  of  equipment  and  information  obtained  from  the  task  analyses 
are  used  to  define  the  task  groups.  The  equipment  configuration  is  obtained 
from  the  engineering  design  process.  The  behavioral  requirements  of  a man- 
machine  task  are  provided  by  the  equipment  in  the  form  of  the  following 
components  (39): 

1.  Displays  that  convey  essential  information  to  the  operator 

2.  Response  alternatives  that  direct  the  operator  to  select  a course  of 
action 

3.  Controls  which  the  operator  must  activate  certain  ways  to  produce 
the  criterion  output  intended  for  the  machine  produces  its  criterion 
output  within  the  quality  tolerances  specified. 

Task  Analysis.  A task  analysis  defines,  systematically  and  in  as  much 
detail  as  necessary,  the  behavioral  requirements  of  the  tasks  to  be  accom- 
plished. It  describes  the  kinds  of  discriminations  that  must  be  made,  the 
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TABLE  4 


Administration 

70 

60 

Task  Group  Description 

160 

130 

Test  Planning  and  Design 

180 

210 

Test  Execution 

100 

100 

Data  Analysis 

120 

80 

Report  Preparation 

130 

140 

TOTAL 

760 

720 
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TABLE  5 


Sample  HFE  Test  Budgets 


HFE  TEST 


COST 

CATEGORY 

EXPERIMENTAL 
PROTOTYPE 
(RPV  Controllers) 

ENGINEERING  DEVELOPMENT 
PROTOTYPE 
(ecu  Operator) 

Labor^ 

$ 9,490.00 

$ 8,300.00 

Mock-up 

Plan 

Design 

Materials 

Fabrication 

$ 900.00 

1.460.00 
440.00 

1.200.00 

4,000.00 

Test  Participants^ 
(50  hrs.  @ $31/h 

r.) 

1 ,550.00 

- 

, Test  Material  & Equipment^ 

1 

150.00 

150.00 

‘ Travel 

600.00 

1 ,800.00 

1 Consultant*^ 

i 

- 

1 ,650.00 

G & A 

1 

2,680.00 

2,020.00 

; Fee/Profit 

1,475.00 

1 ,110.00 

* TOTAL 

1 

$19,945.00 

$15,030.00 

1 ^Burdened  labor 

rate  including  direct  labor 

and  overhead. 

*^i1itary  personnel  were  used  as  test  participants  in  the  CCD  test.  No 
fee  charged  for  their  services. 


‘"Does  not  include  HFE  instrumentation  equipment  which  was  available  for 
use  in  these  tests. 

^Consultant  provided  background  information  on  CCD.  Similar  support  from 
Teledyne  Ryan  Aeronautical  was  provided  at  no  cost. 


decision  making  requirements,  and  the  motor  responses  to  be  made  by  the 
operator.  In  the  present  case,  the  task  analysis  is  used  to  identify  the 
task  group  performance  requirements  to  be  evaluated  in  the  hfe  test. 

Task  definitions  have  been  widely  discussed  in  the  human  factors 
literature  (8,  33,  39).  To  summarize  these  definitions,  a task  may  be  defined 
as  a stimulus-response  relationship.  The  stimulus  is  sensed  by  the  operator, 
is  interpreted  by  him,  and  elicits  a response.  The  completion  of  the  task 
must  satisfy  a specific  requirement  and  it  must  involve  the  output  of  only 
one  man  in  some  combination  with  machine  components.  Above  all,  a task  must 
be  goal  directed.  That  is,  a task  is  a series  of  actions  leading  to  a 
meaningful  outcome. 

The  manner  in  which  tasks  are  described  depends  on  the  task  taxonomy 
selected  and  the  stage  of  system  development.  Many  task  taxonomies  have  been 
developed  (6,  11,  39),  varying  primarily  in  the  level  of  detail  of  the 
analysis.  However,  no  universal  taxonomy  is  dominant  since  the  required 
level  of  detail  depends  on  the  purpose  of  the  task  analysis.  For  the 
purposes  of  the  HFE  test,  the  simple  taxonorny  shown  in  Table  6 is  recommended. 
In  general,  the  taxonon^y  should  conform  to  the  level  of  detail  found  in  the 
stage  of  equipment  development.  For  systems  in  the  experimental  prototype 
phase  of  development,  a two-level  taxonomy  (i.e.,  task  and  subtasks)  should 
be  sufficient.  For  example,  a two-level  taxonomy  corresponded  to  the  level 
of  detail  in  the  experimental  prototype  (RPV  station)  test.  For  the  advanced 
development  prototype  (CCU)  test,  a three-level  taxonony  (i.e.,  task,  subtask, 
and  task  element)  was  used  to  describe  the  operators'  actions. 

Although  there  are  differences  in  emphasis  and  types  of  formats  used, 
all  task  analyses  should  include  the  following  steps: 

1.  Determine  system  functions 

2.  Trace  each  function  to  the  machine  output  or  to  a control 

3.  Determine  display  information  necessary  for  operator  to  decide  to 
activate  a control  or  monitor  a system  state 

4.  Determine  feedback  given  by  machine  to  indicate  appropriate  operator 
response 

5.  Insure  that  each  stimulus  is  linked  to  a response  and  each  response 
to  a stimulus. 

Many  different  formats  have  been  developed  for  recording  and  presenting 
task  analysis  data.  Some  of  the  more  widely  used  techniques  are: 

1.  Task  and  Equipment  Analysis  (TEA) 

2.  Operational  Sequence  Diagrams  (OSD) 

3.  Combined  TEA  and  OSD 

4.  Task  Demands  Analysis. 
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TABLE  6 


Sample  Task  Taxonomy 


FUNCTION 


Activities  assigned  to  person  on  machine  or  between  person  and 
machine  (e.g.,  vehicle  operation). 


*JOB 

The  combination  of  all  human  performance  requirements  (duties  and 
tasks)  or  one  personnel  position  in  a system  (e.g.,  pilot,  driver, 
gunner) . 

DUTY 

A set  of  operationally  related  tasks  within  a given  job  (e.g., 
control  aircraft,  drive  truck,  manage/operate  system). 

*TASK 


A composite  of  related  activities  (perceptions,  decisions,  and 
responses)  performed  for  an  immediate  purpose;  written  in  operator 
language  (e.g.,  takeoff  from  airfield,  prepare  for  departure,  load 
gun). 


SUBTASK 


Activities  (perceptions,  decisions  and  responses)  which  fulfill  a 
portion  of  the  immediate  purpose  within  a task  (e.g.,  monitor  engine 
instruments,  assess  passenger  safety,  position  ammunition). 

TASK  ELEMENT 


The  smallest  logically  definable  unit  of  behavior  required  for 
completing  a task  or  subtask  (e.g.,  verify  RPM  between  4500-6000, 
verify  seat  belts  being  used,  insert  belt  into  feeder). 


*Requlred 
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Several  authors  have  documented  the  procedures  for  performing  these  task 
analyses  (2,  11,  20,  21,  28,  31).  However,  few  specific  guidelines  are 
available  on  what  task  analysis  technique  to  use.  In  general,  use  the 
technique  you  are  most  familiar  with  and  perform  the  analysis  to  the  level 
of  detail  possible  considering  the  stage  of  system  development.  The  TEA  and 
OSD  techniques  provide  methods  for  identifying  tasks  which  must  be  performed 
simultaneously  by  several  operators.  This  is  particularly  true  for  these 
techniques  when  they  include  a time-line  analysis.  These  techniques  can  also 
be  used  to  identify  potential  interference  between  two  or  more  task  groups. 

As  illustrated  in  the  two  sample  HFE  tests.  Task  and  Equipment  Analyses 
were  used  to  derive  the  sequential  lists  of  operator  tasks.  These  lists  of 
tasks  provided  the  basis  for  generating  (1)  the  test  scenarios  which  defined 
the  test  participants'  tasks  and  (2)  the  behavioral  checklists  used  to 
evaluate  their  performance. 

Performance  Standards  Identification.  A man-machine  system  is  an 
organization  whose  components  are  men  and  machines,  working  together  to 
achieve  a common  goal  (19).  In  the  context  of  the  Life  Cycle  System  Manage- 
ment Model,  and  particularly  with  regard  to  the  HFE  test,  the  'system* 
"...consists  of  hardware  (and  sometitres  software);  personnel  who  operate, 
maintain,  and  support  it;  the  training  they  receive;  and  the  tools,  manuals, 
and  equipment  they  use"  (58).  System  requirements  delineate  what  the  system 
must  be  able  to  do:  its  objectives.  Requirements  include  the  mission  or 
purpose  of  the  system  and  the  levels  and  kinds  of  performance  needed  to  meet 
system  goals. 

The  purpose  of  analyzing  these  requirements  is  to  identify  the  specific 
functions  the  system  must  perform.  Functions  are  the  most  general,  yet 
differentiable  means  whereby  the  system  requirements  are  met,  discharged,  or 
satisfied  (8).  Identification  of  functions  leads  to  a determination  of  the 
types  of  human  and  equipment  capabilities  required  to  satisfy  system  require- 
ments. From  a human  engineering  standpoint,  analyzing  system  requirements 
includes  identifying  the  personnel  tasks,  the  specific  task  performance 
standards,  and  the  personnel  selection  and  training  requirements. 

System  requirements  are  almost  always  specified  by  the  customer. 
Requirements  are  first  expressed  qualitatively,  then  quantitatively,  and 
become  increasingly  more  precise  as  system  development  proceeds.  Specific 
equipment  and  personnel  performance  standards  are  typically  developed  after 
functions  have  been  allocated  and  specific  tasks  have  been  determined. 

Human  performance  standards  are  developed  by  working  backwards  from  system 
requirements  and  determining  the  maximum  allowable  task  times  and  error  rates 
acceptable  that  will  still  allow  the  mission  to  be  completed  satisfactorily. 
In  the  case  of  systems  acquired  by  the  government,  new  procurement  policies 
(42)  indicate  that  system  requirements  will  be  given  in  terms  of  required 
system  performance,  rather  than  design  specif iciations.  In  keeping  with  the 
taxonomy  shown  in  Table  6,  the  government  will  thus  specify  the  missions  and 
scenarios  (and  the  associated  performance  standards),  while  the  contractor 
will  specify  the  system  functions,  tasks,  etc.  (and  their  associated 
performance  standards). 
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Several  authors  specify  procedures  for  developing  performance  standards 
(8,  28,  31,  33,  78).  In  many  instances,  system  requirements  are  stated  in 
terms  of  hardware  requirements  with  no  statement  of  human  performance 
influences.  HFE  tests  can  be  conducted  in  the  absence  of  performance 
standards;  however,  the  effect  of  performance  on  overall  system  effectiveness 
can  then  be  expressed  only  with  regard  to  assumed  standards. 


Test  Planning  and  Design 

Test  Objectives  Specification.  Test  planning  begins  with  a statement  of 
the  test  objectives!  In  general,  the  more  precisely  the  test  objectives  are 
defined,  the  easier  it  is  to  develop  the  test  plan.  Since  testing  costs 
money,  the  scope  of  the  test  should  not  only  consider  the  objectives  of  the 
present  test  but  also  what  has  been  determined  from  previous  tests  (56). 
Indeed,  the  Single  Integrated  Development  Test  Cycle  policy  (SIDTC)  clearly 
specified  that  tests  shall  not  be  duplicative.  Thus,  it  should  not  be 
necessary  to  replicate  earlier  test  findings.  For  example,  if  a certain 
sequence  of  tasks  has  already  been  evaluated,  and  hardware  or  procedural 
changes  have  not  been  made  during  the  intervening  period,  it  would  be  a waste 
of  test  time  and  money  to  re-evaluate  these  tasks.  Also,  feasibility  of  task 
performance  is  generally  determined  before  a system  proceeds  to  advanced 
development.  If  task  feasibility  has  been  demonstrated  in  early  development 
testing,  further  time  and  money  should  not  be  spent  re-evaluating  feasibility 
in  the  later  stages  of  development.  Procedures  for  developing  test  objectives 
are  given  in  DI-H-1313A  and  have  been  documented  in  the  literature  (31,  33). 

Test  Equipment  Design  and  Selection.  The  nature  of  the  HFE  test  equip- 
ment configuration  depends  upon  the  stage  of  system  development.  Specific 
task  group  performance  questions  are  answered  most  cost-effectively  at 
different  stages  of  system  development.  Control  and  display  arrangement, 
man-machine  function  allocation,  and  accessibility  of  components  for  main- 
tenance, etc.,  are  determined  most  effectively  before  any  prototype  hardware 
is  built.  Mock-ups  should  be  used  to  answer  these  types  of  questions.  For  a 
test  of  a system  in  experimental  prototype  development,  a mock-up  will  often 
be  required.  As  system  development  progresses,  breadboard  and  brassboard 
prototype  equipment  will  be  included  in  the  test  station. 

Specific  questions  on  the  ability  of  the  task  group  to  meet  or  exceed 
performance  specifications  are  best  answered  on  operational  equipment. 
Sophisticated,  dynamic  computer-controlled  mock-ups  and  simulators  can  also 
be  used  as  test  equipment  in  the  later  stages  of  system  development.  Dynamic 
mock-ups  and  simulators  should  be  used  only  when  they  are  the  most  cost- 
effective  method  of  answering  questions  critical  to  mission  success.  For 
example,  simulators  are  often  used  in  flight  training  because  it  is  cheaper 
and  safer  to  train  pilots  in  simulators  than  it  is  to  train  them  in  aircraft. 
The  literature  also  contains  procedures  for  developing  and  evaluating  mock- 
ups  (5,  45,  78). 

Test  Enyiroment  Measurement.  An  HFE  test  must  take  into  account  the 
environmental  conditions  under  which  the  personnel  and  equipment  are  expected 
to  be  used.  However,  to  be  cost-effective,  the  HFE  test  need  simulate  only 
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those  environmental  conditions  which  are  likely  to  affect  task  group  per- 
formance. Objective  (previous  tests  or  analysis  of  relevant  performance 
literature)  or  subjective  (interviews  or  questionnaires  of  subject-matter 
experts)  methods  should  be  used  to  determine  the  environmental  conditions  to 
be  simulated  and  measured.  The  Materiel  Test  Procedure  (MTP)  (63-75)  for  the 
class  of  equipment  being  tested  should  be  reviewed  to  determine  the  critical 
measurements  to  be  taken.  Major  environmental  measurement  areas  that  should 
be  considered  are  also  shown  in  Table  7. 

The  degree  to  which  the  test  environment  simulates  operational  conditions 
depends  upon  the  stage  of  system  development.  Normally,  it  is  not  cost- 
effective  to  allocate  substantial  resources  to  simulate  all  expected  use 
conditions  during  the  early  stages  of  development.  For  example,  even  if 
vibration  is  expected  to  affect  task  group  performance,  little  would  be  gained 
by  simulating  vibration  when  the  participants  are  being  tested  on  a static 
mock-up  since  the  effects  of  these  environmental  conditions  should  be 
estimated  by  analysis  and  by  reference  to  previous  experimental  results. 
Environmental  "realism"  should  be  incorporated  in  tests  in  the  later  stages 
of  system  development.  The  effects  of  environmental  conditions  on  performance 
are  best  assessed  on  dynamic  mock-ups,  simulators,  and  on  prototype  equipment 
that  include  all  the  critical  environmental  conditions  of  the  field-deployable 
system. 

After  the  required  environmental  conditions  have  been  specified  for  the 
HFE  test,  provisions  must  be  made  for  measuring  these  critical  conditions 
during  the  test.  If  the  test  environment  is  stable,  the  environmental 
measurements  need  be  recorded  only  once  during  the  test  period.  However, 
changing  environmental  conditions  will  necessitate  more  frequent  measurement. 
Table  8 lists  the  measurement  instruments  of  the  Human  Factors  Instrumentation 
Package  (61)  which  can  be  used  to  measure  the  environmental  conditions.  This 
HFE  Instrumentation  Package  contains  all  of  the  instruments  required  to  obtain 
the  environmental  measurements  recorded  in  the  two  sample  HFE  tests.  To 
reduce  test  costs,  a contractor  may  be  able  to  borrow  an  HFE  Instrumentation 
Package  from  the  human  engineering  unit  at  the  Arn\y  command  which  issues  the 
system  development  contract.  A comprehensive  summary  and  description  of 
environmental  recording  techniques  and  procedures  are  contained  in  the 
reference  manual  for  the  Human  Factors  Instrumentation  Package.  Environmental 
measurement  techniques  are  also  discussed  in  the  Human  Factors  Test  and 
Evaluation  Manual  (24)  and  in  selected  Materiel  Test  Procedures  (63,  70,  71, 
75). 


Test  Participant  Selection.  To  a large  extent,  the  validity  of  the  HFE 
test  results  depends  on  how  well  the  human  factors  engineer  is  able  to  match 
his  test  participants  to  the  characteristics  of  the  personnel  who  will 
ultimate-ly  use  the  equipment.  In  early  stages  of  system  design,  participants 
are  usually  drawn  from  contractor  technicians,  engineers,  and  other  company 
personnel.  Contractor  personnel  are  often  more  skilled  and  experienced  than 
the  military  personnel  who  will  ultimately  use  the  system,  and  this  disparity 
can  sometimes  lead  to  erroneous  conclusions.  For  example,  highly  skilled 
personnel  may  not  be  aware  of  difficulties  which  would  be  severe  if  less 
skilled  personnel  were  using  the  equipment.  However,  the  advantage  of  using 
highly  skilled  personnel  in  the  early  stages  of  development  cannot  be  ignored. 
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Test  Measurement  Categories 
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MOTE:  T«ble  entries  ere  specific  eeasurewents. 

Muiters  In  parentMeses  refer  to  Instn^ent  list  (Table  8). 


TABLE  8 


Measurement  Equipment  List 


HCASUKMNT  ARCA 

instmmnt 

COMPONENTS  ANO/OR 

ACCESSORIES 

MANUFACTURER 

NOISE 

•n4 

VIBRATION 

1 

SowAd  L*««l 

Mtttr/AMlyitr 

Micropttena.  aatantion  cabla. 

AC  adaptor/cNargar.  carrying  caaa 

Tripod 

(Wnaral  Radio 

300  baker  Avenue 

Concord,  Massarhusatts  01TA2 

z 

VINratlQA  Ltvtl 

PIttcr  Analyitr 

VipratlOA  pick-up  and  aount. 
raaota  aaaturing  package 

lUlRIINATION 

•n4 

BRIBHTNCSS 

3 

PAotOMtA' 

Sensor  proPa,  11  liartnatton  and 
luainancB  racapters,  carrying  case 

Rhoto  Research  Division 
(oilnorgan  Corporation 

3000  North  HoHyHoed  Ray 
Burbonk.  CallForvila  9IS0S 

4 

Spot  BrIfhtAOtS  Nct«r 

Closa-up  Ians,  battery  pack 

FORCE 

•Ad 

OIPtCNSION 

FOftt.  Torqu*.  iAfl 

OlaOAtlOA  Kit 

Dial  torgua  gauge,  dial  Force  gauge 
Pusb/Puii  gauge  ill.  4iai  torgua 
urtncH  (2),  torgua  adaptor,  socket 
sat,  adj . protractor,  tapa  Measures  (2) 

— 

Chatlllon 

B3-30  Rau  Gardens  Road 

Kan  Gardens.  Nan  Tort  11419 

ATMOSPHERIC 

ENVIROfMEN' 

‘ 

PortaPit  Maatnar 

StatloA 

Rind  spaed  and  direction,  taa^eraiura. 
M^ldlty  and  baroMtrlc  pressure  aatars 

Rind  sensor  (vane) 

■aathar  Measure  Corporation 

R.O  Boi  4I2S7 

Sacrpapnto.  CallFernia  gSRTl 

1 

Hot  Wirt  Anoaomtar 

Ranota  probe,  carrying  case 

« 

AspIratlA^  PtycAroBKtar 

PsycbrgMtrlc  slide  rule 

9 

Instriavnt  aUA  built-in  battery 
pack  cbargar 

Oigitac.  Inc. 

911  Woodley  Road 

Dayton,  Ohio  4S403 

RfMota  orobas  (surface,  air.  oral) 

POILUTANTS 

,0 

UAiyartal  Gat  Tattar 

Datactor  tubas  (CO.  C02>  RD;. 
and  SO?),  hand  puap.  rgnota 
saMplIng  tuba,  carrying  case 

Nina  SaFaty  ApoHancat  Co 

400  Rann  Central  Boulevard 
RlttSburgh,  Rannty1van1a1S23& 

NoAlterlA)  Gas  Saaplar 

Oust  celipctor.  gas  lapingar. 
battery  pack/chargar 

ANThROPOMCTRv 

)? 

Arththropoaatry 
lAttruHOAt  Kit 

AnthropOMatar.  tape,  sliding 
caliper,  spreading  caliper, 
carrying  case,  gonionatar, 
eaigh  scale 

Sibar  Rrtclslon,  Inc. 

490  Bareli  Avenue 

Carlstadt.  nm  Jersey  07072 

PERFORMANCE 

13 

Digital  Tlwr 

Ranota-control  svitch,  clipboard 

AC  adaptor/chargar . carrying  case 

Heath  Coapanv 

Benton  Harbor 

Michigan  4go2? 

14 

Multiplt  CvoAt  Countar 

Digital  counter  (3) 

RuAh-button  suHch  (3) 

Rbotocall  sintch  (!) 

Raugh  Controls  Corporation 

7621  Hayvanhurst  Avenue 

Van  Nuyi.  CaHFomla  91406 

IS 

Polaroid  SI70  Cawra 

Clesa-up  Ians.  Flash  bar. 
tripod,  carrying  case. 

Rolarold  Corporation 

730  Main  Street 

Caabridga.  Natsaciiusatti02i31 

16 

vioaetapa  Raceroing  Syttaai 

COMPra.  racordar,  aBnltor,  lOO*  Ians 

AC  adaptor/cNprgar.  carrying  caaa 

Eitanitan  cabla 

Tripod 

Ranasonic 

200  Rark  Avenue 

Na*  York,  Na«  Tan  10017 

RECORDING 

•Ad 

ANALtSIS 

17 

Audio  Tapa  Racordar 

Raneta  nlcroptwna. 

AC  adaptor/chargar, 
carrying  case,  aarphona 

Sony , Inc 

2160  Vineland  Avenue 

Sun  valley.  California  91392 

IR 

InttriaaoAtatlOA 

Tapa  Racordar 

MicropNena,  compctor  cables, 
battery  pack/chargar.  aarphon# 

Recorder,  carrying  caaa 

Toac  Corporation  of  Raprica 
7733  Toiograpn  Road 

Montebello.  CpIlForrla  90640 

19 

SciantlFlc  Calculator 

RC  adaptor/chargar  aoFt 
carrying  caaa,  travel  case 

Neulatt-Rackard,  Inc 

10900  RolFa  Road 

Cupertino,  CaliFenila  99014 

maintenance 

•a4 

calibration 

M 

Digital  Tost  Maiar 

Raaata  probes,  AC 
adaptor/chargar. 
carrying  caaa 

Ratten  Instrtfvnts.  Inc. 

614  Frtlingbuysan  Avenue 

NatMrk  . Ne«  Jersey  07144  ' 

?1 

Tool  Kit 

Assorted  icrpedrivars,  ngt  drivers 

and  pliers. 

Xcallta,  Inc. 

770  Bant  St 

Orchard  Rark,  N.t  14127  • 

22 

Battary  CNargar 

1 

General  Electric  Coapan,  ^ 

1 
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Such  skilled  personnel  can  provide  detailed  information  on  task  sequencing 
and  definition.  They  can  also  provide  information  on  the  adequacy  of  the 
control-display  layout  (For  example,  are  all  of  the  required  controls  and 
displays  present,  and  are  any  unnecessary  controls  included,  etc,?). 

Although  the  practice  of  using  contractor  personnel  as  test  subjects  is 
common,  simple,  and  expedient,  it  almost  invariably  results  in  biased 
outcomes  (78).  This  practice  of  using  overly-skilled  personnel  should  be 
limited  to  testing  in  the  earliest  stages  of  development.  The  importance  of 
using  representative  personnel  as  test  participants  increases  as  a system 
nears  completion.  Geddie  (12)  provides  a good  description  of  the  methods  and 
criteria  to  be  used  in  selecting  test  participants  for  developmental  tests. 

The  advantages  of  using  military  personnel  include  the  following: 

1.  Military  personnel  are  usually  more  representative  of  the  user 
population  than  are  contractor  personnel.  When  selected  from 
intended  user  organizations,  military  personnel  will  be  represent- 
ative in  terms  of: 

a.  Skill  level  (similar  Military  Occupation  Specialty) 

b.  Motivation  (no  particular  incentive  to  make  equipment 
"look  good") 

2.  Testing  costs  can  be  reduced  since  Single  Integrated  Development 
Test  Cycle  (SIDTC)  (56)  policy  envisions  use  of  government-supplied 
military  test  participants, 

3.  Intelligence  testing  can  be  eliminated  since  Army  General  Classif- 
ication Test  (AGCT)  scores  are  available  for  military  test 
participants. 

DI-H-1334A  includes  several  requirements  to  assure  that  the  selected  test 
participants  are  representative  of  the  user  population.  These  requirements 
include  the  minimum  numbers  of  participants  used  in  the  HFE  test,  the 
descriptions  of  the  participants  (12),  and  the  training  of  participants. 

Both  the  number  and  skill  levels  of  the  test  personnel  must  be  determined 
prior  to  testing.  In  general,  the  more  participants  tested,  the  more  reliable 
the  test  results.  However,  the  cost  of  conducting  the  test  increases 
monotonically  as  the  number  of  test  participants  increases.  Tradeoffs  must 
be  made  between  data  reliability  and  test  costs,  DI-H-1334A  requires  at 
least  three  participants  per  task  group.  The  effect  of  test  participant 
performance  variability  is  an  important  factor  in  determining  the  number  of 
test  participants.  In  experimental  prototype  tests,  participant  variability, 
in  terms  of  time  requirements  for  monitoring  static  displays  and  adjusting 
static  controls,  may  reflect  test  participant  differences  more  than  it  does 
actual  task  response  times.  Thus,  the  minimum  number  of  personnel  required 
by  DI-H-1334A  (three  participants)  is  generally  sufficient  to  obtain  the 
required  frequency  and  sequence-of-use  data  derived  from  a static  mock-up. 
Performance  times  and  error  rates  obtained  in  tests  of  systems  in  advanced 
and  engineering  development  are  more  representative  of  operational  performance 
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than  earlier  test  results.  Therefore,  a more  precise  estimate  of  performance 
may  justify  a larger  sample  size  to  reduce  the  measurement  confidence  limits. 
In  general,  four  to  six  test  participants  should  be  sufficient  to  obtain 
reliable  results  at  a reasonable  cost.  In  the  sample  HFE  tests,  three 
participants  were  used  in  the  experimental  prototype  (RPV)  test,  whereas 
five  participants  were  used  in  the  engineering  development  prototype  (CCD) 
test. 


To  provide  indices  of  the  representativeness  of  the  test  participants 
as  compared  with  the  eventual  user  population,  DI-H-1334A  requires  that 
measures  of  intelligence  and  other  personnel  characteristics  be  obtained  and 
reported.  If  the  test  participants  are  civilian  operators,  professionally 
accepted,  general  intelligence  tests  are  required  (4,  7).  The  objective  of 
such  intelligence  testing  is  to  determine  the  representativeness  of  the 
sampled  participants,  rather  than  to  measure  exhaustively  the  participants' 
range  of  intelligence.  Therefore,  one  or  more  short,  paper-and-pencil  tests 
of  general  intelligence  should  be  quite  sufficient.  In  the  sample  experi- 
mental prototype  test  where  civilian  operators  were  used,  two  well-established 
psychological  tests  (Otis-Lenon  and  Bennett  Mechanical  Comprehension)  were 
used  to  test  general  intelligence.  If  military  personnel  are  used  as  test 
participants,  the  Verbal  Reasoning  and  Mathematical  Aptitude  scores  of  the 
Army  General  Classification  Test  (AGCT)  may  be  used. 

Additional  measures  required  by  D1-H-1334A  include  age,  weight,  body 
dimensions,  visual  acuity,  and  physical  disabilities.  The  degree  of 
precision  required  in  these  measurements  is  dictated  by  operator  tasks 
identified  in  the  task  analysis.  If  the  operator's  tasks  require  extensive 
visual  skills,  additional  tests  of  visual  capabilities  (e.g.,  depth  perception, 
color  perception,  etc.)  may  be  required.  However,  for  the  majority  of  tasks, 
a simple  measure  of  acuity  (e.g.,  Snellen  eye  chart  or  Armed  Forces  Clinical 
Visual  Acuity  Test)  should  be  sufficient.  Procedures  for  measuring  visual 
capabilities  have  been  documented  (14,  23,  51).  Similarly,  only  the  body 
dimensions  relevant  to  the  successful  and  comfortable  execution  of  the 
identified  tasks  should  be  recorded.  Techniques  for  anthropometric  measure- 
ment are  given  by  Roebuck  (45)  and  VanCott  and  Kinkade  (78). 

Test  Participant  Training.  Equipment,  no  matter  how  appropriate  for  the 
given  task,  cannot  be  adequately  evaluated  unless  test  participants  are 
suitably  trained.  Because  untrained  test  participants  can  make  a system  look 
worse  than  it  is,  DI-H-1334A  includes  requirements  to  assure  that  the  test 
participants  are  trained.  The  participants  must  be  informed  of  their  test 
duties,  and  their  knowledge  of  the  equipment  and  their  tasks  must  be  assessed 
prior  to  testing.  Pre-test  assessment  of  training  assures  that  the  test 
participants  are,  indeed,  sufficiently  prepared  and  trained  for  the  test. 
Additionally,  the  pre-test  training  a'jsessment  data,  in  conjunction  with  the 
test  data,  can  be  valuable  in  estimating  training  requirements  for  the 
expected  user  personnel . 

Assessment  of  test  participants'  abilities  and  knowledge  is  a critical 
concern  of  the  developmental  and  operational  testers,  and  is  one  of  the 
major  control  features  built  into  D1-H-1334A.  Preassessing  test  participant 
abilities  and  knowledge  provides  information  on  whether  the  participants  can 


perform  the  required  tasks  adequately  with  the  training  provided  along  with 
providing  baseline  data  for  later  testing.  Pre-test  knowledge  is  assessed 
by  administrating  paper-and-pencil  achievement  tests  and/or  by  having 
operators  perform  required  tasks  on  the  test  equipment.  Items  for  the  tests 
can  be  obtained  from  previous  training  tests,  if  any  exist,  or  new  questions 
can  be  derived  from  the  task  analysis.  As  an  example.  Appendix  1 contains 
the  achievement  test  developed  for  the  CCU  training  test.  Procedures  for 
developing  training  test  questions,  job  sample  tests,  and  for  evaluating 
training  programs  are  given  in  various  texts  (3,  14,  51)  and  selected  MTP's 
(64,  69,  71). 

The  type  and  amount  of  pre-test  training  depend  on  the  stage  of  system 
development  and  the  uniqueness  of  the  system.  For  a system  in  early  develop- 
ment, or  for  a system  that  is  a radical  departure  from  earlier  systems, 
trained  operators  will  not  be  available.  In  these  instances,  thorough  pre- 
test indoctrination  will  be  required  to  introduce  the  participants  to  the 
equipment,  operational  procedures,  and  task  requirements.  For  tests  of 
systems  in  advanced  stages  of  development,  trained  operators  should  be  more 
readily  available.  In  this  latter  case,  pre-test  training  will  consist  of 
briefings  on  test  procedures  and  performance  requirements,  and  then  a period 
of  controlled  practice. 

Data  Acquisition  and  Analysis  Planning.  Both  objective  and  subjective 
data  collection  techniques  are  required  to  obtain  the  information  required 
by  DI-H-1334A.  The  distinguishing  feature  between  the  two  is  the  source  of 
information  obtained.  Information  which  depends  on  judgement  or  opinions  of 
test  engineers  and/or  test  participants  is  classified  as  subjective.  If  the 
source  of  information  entails  measurement  of  man-machine  system  character- 
istics or  performance,  then  the  method  is  considered  objective.  Examples  of 
subjective  methods  include  ratings,  rankings,  questionnaires,  and  interviews. 
Objective  methods  include  measurement  of  performance  time,  error  rates,  event 
frequencies,  as  well  as  the  determination  of  dimensions,  illumination  levels, 
forces,  noxious  gases,  etc. 

In  general,  objective  measures  should  be  employed  as  much  as  possible, 
due  to  the  following  advantages: 

1.  The  degree  to  which  performance  standards  are  met  may  be  determined 

2.  Comparison  of  the  obtained  measurements  with  the  criteria  does  not 
require  subjective  inference 

3.  The  results  are  repeatable  since  error  arising  from  human  judgment 
is  minimized. 

Objective  methods  are  most  appropriate  for  answering  two  of  the  main  questions 
posed  by  DI-H-1334A:  the  feasibility  of  task  group  performance  and  the 
adequacy  of  the  man-machine  interface.  Objective  performance  assessment 
procedures  are  described  by  several  authors  (5,  8,  24,  40,  78)  and  in 
MIL-HDBK-759  (35). 
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A good  technique  of  objective  performance  assessment  is  direct  observa- 
tion by  a trained  HFE  test  observer  who  uses  a behavioral  checklist  (27)  to 
assure  that  all  of  the  required  tasks  are  performed  and  to  provide  an  orderly 
way  of  recording  data.  With  this  technique,  an  observer  is  required  for  each 
tested  personnel  position;  and  a different  checklist  will  be  used  for  each 
personnel  position.  The  checklist  is  also  used  to  record  on-site  comments 
and  error  descriptions.  Appendix  2 contains  a checklist  used  in  one  of  the 
test  segments  of  the  CCD  sample  HFE  test. 

Greater  reliability  can  be  obtained  by  using  objective  observation 
techniques,  supplemented  by  audio,  video,  and  film  monitoring.  Audio 
recordings  car  be  used  to  docment  sequence  errors  and  task  group  communi- 
cations. Video  recordings  ar  particularly  useful  in  studying  human 
performance  to  determine  causes  of  errors,  equipment  incompatibilities  and 
task  group  interference  problems.  Video  recordings  can  also  be  edited 
(significant  events  can  be  highlighted)  and  used  for  presentations  at  design 
review  meetings,  etc. 

Automated  data  collection  and  analysis  techniques  can  also  be  used, 
particularly  with  a dynamic  mock-up  that  is  controlled  by  a computer. 

Automated  techniques  are  often  capable  of  providing  considerably  more 
accurate  information  than  can  be  collected  or  analyzed  within  the  test 
constraints.  Automated  data  collection  may  be  particularly  useful  in 
analyzing  time  and  sequence-of-use  events.  However,  the  time  required  to 
develop  the  necessary  data  reduction  and  analysis  software  must  be  considered. 
If  repetitive  testing  is  envisioned,  such  software  developments  can  be 
particularly  cost-effective. 

While  objective  measurements  may  indicate  whether  criteria  are  met,  they 
may  not  provide  an  adequate  explanation  for  substandard  performance.  It  is 
here  that  subjective  methods  may  be  a valuable  supplement  for  determining  the 
causes  of  such  problems.  The  person  who  operates  or  uses  an  item  has 
firsthand  knowledge  of  its  operating  characteristics.  Consequently,  evalu- 
ations should  take  advantage  of  the  subjective  opinions  and  observations  of 
those  who  use,  operate,  and  maintain  equipment.  To  collect  these  observations, 
carefully  planned,  structured,  face-to-face  interviews  are  preferred. 
Questionnaires  can  also  be  used,  or  the  two  techniques  may  be  used  together. 

Again,  objective  methods  alone  are  insufficient  for  gathering  data  on 
the  adequacy  of  personnel  training  and  selection  criteria.  Subjective  methods 
also  are  needed  whenever  it  is  necessary  to  obtain  such  information  as  the 
degree  of  user  acceptability,  motivation,  comfort,  or  convenience. 

Questionnaires  and  interviews  help  determine  the  causal  factors  to  which 
system  inefficiency  can  be  related.  Since  the  determination  of  causal  factors 
is  the  only  basis  upon  which  corrective  action  can  be  taken,  the  data  gained 
from  the  questionnaires  and  interviews  are  an  extremely  important  part  of  the 
total  data  pool.  However,  since  corrective  action  will,  in  many  cases,  have 
a high  cost  asssociated  with  it,  care  must  be  taken  in  the  administration  and 
subsequent  interpretation  of  the  data.  While  the  data  from  the  questionnaires 
and  interviews  will  be  supplemented  by  objective  performance  data,  these  data 
must  also  be  able  to  stand  alone  as  a useful  data  source.  Appendix  3 contains 
an  example  questionnaire  that  was  used  in  the  sample  tests. 
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Data  obtained  from  interviews,  checklists,  and  questionnaires  are  also 
useful  in  determining  and  analyzing  test  problems  and  errors.  A sample  HFE 
checklist  is  contained  in  Appendix  4.  Interviewers  and  questionnaires  should 
ask  the  test  participants  to  describe  and/or  list  the  problems  they  exper- 
ienced during  testing  and  how  they  think  their  working  environment,  tasks, 
etc.  could  be  changed  to  improve  their  performance.  Information  on  potential 
safety  hazards  should  also  be  solicited.  All  of  these  techniques  for  record- 
ing data  complement  one  another  in  compiling  the  data  needed  for  developing 
complete  narrative  descriptions  of  test  problems  and  errors.  In  addition  to 
the  previous  references  given  for  objective  assessment  techniques,  a further 
disucssion  of  subjective  techniques  has  been  recently  published  (62). 

As  stated  in  DI-H-1334A,  the  HFE  test  serves  as  the  principal  means  of 
substantiating  the  feasibility  of  required  human  performance,  the  accuracy 
of  the  personnel  selection  criteria,  the  adequacy  of  the  training  program, 
and  the  acceptability  of  the  man-machine  interface.  Therefore,  the  data  to 
be  collected  must  be  useful  in  responding  to  these  requirements.  The  task 
analysis  identifies  all  of  the  tasks  that  the  operator  must  perform.  The  data 
which  are  collected  must  indicate  the  time  required  to  complete  each  task  and 
must  indicate  the  type  and  amount  of  errors  in  per^'orming  these  tasks. 

Data  must  also  be  gathered  about  human  performance  that  is  critical  to 
mission  success.  Critical  human  performance  is  that  which,  if  not  accom- 
plished in  accordance  with  system  requirements,  will  have  adverse  effects  on 
cost,  system  reliability,  efficiency,  effectiveness,  or  safety.  Human 
performance  is  considered  critical  whenever  the  test  item^s  characteristics 
demand  performance  which  challenges  human  capabilities — and  which  may 
therefore  contribute  significantly  to  conditions  such  as: 

1.  Jeopardizing  performance  of  a mission 

2.  Delaying  a mission  beyond  acceptable  time  limits 

3.  Causing  improper  operation  that  produces  a system  "no-go," 
inadvertent  weapons  firing,  or  failure  to  achieve  operational 
readiness 

4.  Exceeding  predicted  times  for  maintenance  tasks 

5.  Degrading  system  equipment  below  reliability  requirements  (mean 
time  between  failure  (MTBF)  is  reduced) 

6.  Damaging  system  equipment  so  it  must  be  returned  to  a maintenance 
facility  for  major  repair,  or  causing  unacceptable  costs,  spare 
requirements,  or  system  downtime 

7.  Seriously  compromising  weapon  system  security 

8.  Injuring  personnel. 

Any  evaluation,  even  one  consisting  only  of  questionnaire  or  interview 
data,  will  involve  some  sort  of  statistical  summary  and  analysis.  The 
statistical  treatment  may  be  relatively  simple,  involving  no  more  than  some 
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summary  tables  with  averages  or  some  other  descriptive  statistics,  or  it  may 
be  quite  complex,  encompassing  curve  fitting  or  analysis  of  variance. 

Whatever  its  nature  or  form,  a statistical  analysis  can  be  no  better  than  the 
data  that  enter  into  it.  For  this  reason,  the  choice  of  data  reduction 
procedures  must  be  developed  prior  to  testing  and  must  be  compatible  with  the 
data  to  be  collected.  When  analysis  procedures  are  established  prior  to 
testing,  the  test  personnel  can  insure  that  all  of  the  data  required  for  the 
selected  analysis  are  obtained  during  the  testing.  Several  authors 
(5,  28,  30,  47,  79)  describe  data  analysis  techniques  for  the  various  types 
.of  data  obtained  from  a DI-H-1334A  application.  Analysis  techniques  are 
'also  described  in  the  Data  Analysis  section  of  this  report. 

Test  Segment  Development.  The  operator  tasks  that  were  identified  in 
the  task  analysis  must  be  scheduled  so  they  occur  in  proper  sequence  during 
the  test  period.  For  some  systems,  the  machines'  operational  sequence 
dictates  the  sequence  of  tasks.  For  example,  in  the  sample  test  of  the  RPV 
control  station,  the  vehicle's  flight  stage  determines  the  order  of  control 
operations.  Thus  the  task  scenario  was  developed  so  it  followed  this  natural 
sequence  of  operations.  For  other  systems,  tasks  may  be  performed  in  ir- 
regular sequences,  as  with  the  CCU  in  the  TACFIRE  system.  Here,  the  operator' 
task  scenario  was  designed  so  it  included  all  identified  tasks.  In  both 
cases,  the  scenario  included  all  tasks  that  had  been  identified  in  the  task 
analysis.  Appendix  5 presents  a sample  test  scenario. 

Where  the  preliminary  hazard  analysis  of  the  system  has  identified 
potential  safety  problems,  the  test  manager  should  insure  that  the  test 
segments  planned  include  performance  of  all  tasks  related  to  the  hazards 
identified  and  that  appropriate  data  (at  minimum,  the  time  and  error  dimen- 
sions of  human  performance;  frequently,  measures  of  equipment  performance  or 
environmental  characteristics)  will  be  obtained  from  which  to  determine 
whether  each  hazard  predicted  actually  occurs  and,  if  so,  whether  adequate 
safety  measures  exist  to  minimize  its  effect(s). 

The  test  scenario  can  be  written  in  a manner  similar  to  a play  script. 

To  minimize  paper  shuffling  during  test  execution,  these  scenarios  should  be 
combined  with  the  behavioral  checklists.  The  behavioral  checklist  includes 
the  description  of  the  task  to  be  performed  (scenario),  the  indication  of 
correct  performance,  and  columns  for  recording  performance  times  and  errors. 
Use  of  these  checklists  substantially  reduced  the  effort  required  to  collect 
human  performance  data  in  the  sample  tests. 

Test  Schedule  Preparation.  A schedule  that  specifies  the  segments  and- 
the  operators  to  be  evaluated  at  each  test  session  should  be  developed  prior 
to  testing.  If  the  schedule  is  followed,  problems  associated  with  what 
participants  and  segments  are  to  be  evaluated  during  any  particular  session 
should  be  minimized.  Time  estimates  for  completing  all  of  the  tasks  required 
for  each  of  the  major  segments  of  a task  group  are  derived  from  the  task 
analysis,  from  system  designers,  experienced  personnel,  or  from  performance 
standards,  if  they  exist. 

Test  Dry  Run  Execution.  After  the  test  has  been  designed  and  all 
sessions  have  been  scheduled,  a preliminary  run-through  should  be  performed. 
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The  purpose  of  this  dry  run  is  to  evaluate  the  test  to  assure  that  all  data 
can  be  recorded  and  that  the  test  moves  smoothly.  For  this  preliminary  test 
evaluation,  substitute  experienced  operators  or  HFE  test  personnel  for  the 
test  participants.  This  evaluation  is  used  to  assure  that  all  tasks  have 
been  included  and  properly  sequenced.  This  preliminary  evaluation  will 
identify  potential  test  deficiencies  and  minimize  false  starts.  However, 
this  preliminary  evaluation  is  not  used  to  collect  test  data. 


HFE  Test  Exeuction 

Pretest  Procedures.  Before  the  test  is  conducted,  several  activities 
must  occur  with  the  test  participants.  A pretest  session  provides  an 
opportunity  to  obtain  biographical,  anthropometric,  and  descriptive  data  from 
the  participants  (12).  Pretest  training  and  practice  can  also  be  given  during 
this  same  period.  Training  evaluation  testing  should  be  completed  before  the 
HFE  test  is  started  to  assure  that  all  participants  have  been  sufficiently 
trained  to  perform  all  required  tasks.  This  training  evaluation  test  may 
include  a sample  of  the  participants'  performance  on  selected  portions  of  the 
HFE  test  scenario.  A performance  sample  supplements  the  job  knowledge  test 
in  assessing  a participant’s  readiness  for  testing.  Criteria  for  accepting 
a participant  for  the  HFE  test  include  on-the-job  knowledge  and  performance 
sample  test  scores,  as  well  as  the  opinion  of  expert  operators  or  instructors 
that  the  participant  is  ready  for  testing  and  motivated  to  participate  in  it. 

Test  Execution.  The  test  is  conducted  according  to  the  plan  and 
sequences  developed  earlier.  The  test  will  usually  require  a minimum  of  two 
test  personnel  to  direct  the  test  activities,  to  observe  test  participant 
performance,  including  performance  times,  errors,  and  task  group  interference, 
and  to  record  performance  data. 

The  environmental  measures  should  be  recorded  as  often  as  scheduled  in 
the  test  plan.  Periods  between  successive  participants  provide  convenient 
opportunities  to  record  these  measures.  The  final  test  session  will  include 
any  questionnaires  and  interviews  designed  to  obtain  subjective  measures  of 
task  group  performance,  and  acceptability  of  the  environment,  adequacy  of 
training,  etc.  To  preclude  test  biases,  previously  tested  participants  should 
not  be  allowed  to  discuss  the  test  with  untested  personnel. 


Data  Analysis 

Olajective  and  Subjective  Data  Summarization.  In  the  experimental  proto- 
type phases  of  system  testing,  the  HFE  tests  focus  on  determining  the  adequacy 
of  the  man-machine  interface,  the  appropriateness  of  task  allocations  to  man 
and  machine,  and  the  feasibility  of  task  group  performance.  In  that  stage  of 
development,  the  HFE  data  analysis  focuses  on  sequence,  frequency  of  use,  and 
criticality  of  equipment  components.  Reliance  on  time  and  error  rate  data 
depends  on  the  degree  of  realism  in  the  simulation  of  control  movements  and 
display  indications.  For  example,  in  a test  of  a static  mock-up,  the  times 
required  and  the  errors  observed  in  actuating  and  adjusting  controls  and 
monitoring  static  displays  may  bear  little  resemblance  to  ultimate  operational 
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times  and  errors.  However,  time  and  error  data  should  be  obtained  to  provide 
baseline  rates  for  later  test  evaluations.  To  enhance  the  usefulness  of  these 
data  for  comparisons  with  subsequent  data,  measured  performance  times  should 
be  stated  such  that  statistical  treatments  may  be  applied.  Where  the  number 
of  test  participants  is  small,  the  raw  data  may  be  stated  (as  in  Table  26). 
Otherwise,  mean,  standard  deviation,  and  sample  size  should  be  reported.  For 
error  data,  both  the  number  of  observed  errors  and  the  total  number  of 
opportunities  for  error  should  be  stated  for  each  task.  In  static  mock-up 
tests,  it  is  more  appropriate  to  assess  the  interface,  task  allocation,  and 
task  feasibility  indirectly  by  analyzing  the  design  of  the  work  space.  Link 
analysis/work  process  charts  may  be  used  to  derive  frequency  and  sequence  of 
use  of  all  controls  and  displays  used  by  the  task  group  (see  Table  14  for  a 
sample  of  a work  process  chart).  Psychological  scaling  techniques  (rating, 
rank  order,  pair-comparison,  etc.)  are  useful  in  determining  the  importance/ 
criticality  of  all  required  equipment.  Scaling  data  can  be  combined  with 
frequency  and  sequence-of-use  data  to  determine  whether  controls  and  display 
arrangement  criteria  (sequence  and  frequency  of  use,  criticality,  common 
functions,  etc.)  have  been  satisfied  (36).  Procedures  for  performing  link/ 
work  process  analyses  have  been  described  (5,  8),  as  have  psychological 
scaling  techniques  (13,  18,  22). 

Anthropometric  suitability  of  the  man-machine  interface  is  assessed  by 
comparing  the  layout  of  the  work  space  to  the  standards  contained  in 
MIL-STD-1472B  (36).  The  literature  also  contains  procedures  for  evaluating 
anthropometric  suitability  (35,  45,  78). 

For  HFE  tests  in  advanced  and  engineering  development,  the  adequacy  of 
the  man-machine  interface  and  human  performance  reliability  and  efficiency 
can  be  assessed  directly  by  analyzing  the  obtained  performance  time  and  error 
data.  Direct  time  and  error  rate  assessment  also  apply  to  experimental 
prototype  tests  conducted  on  equipment  or  mock-ups  that  closely  simulate 
operational  equipment. 

Data  analysis  techniques  for  determining  the  adequacy  of  the  personnel 
selection  criteria  and  the  training  program  range  from  performing  content 
analysis  of  interviews  and  questionnaires  to  summarizing  personnel  task 
times  and  error  rates.  Content  analysis  techniques  are  appropriate  for  both 
experimental  prototype  and  advanced  development  tests.  Dunnette  (9), 

Guilford  (13),  and  Guion  (14)  describe  interview  and  questionnaire  surtmary 
techniques.  The  observation  of  personnel  response  times  and  error  rates  is 
most  appropriate  during  advanced  and  engineering  development.  Inadequacies 
in  training  and  selection  criteria  are  demonstrated  by  inappropriately  long 
response  times  and  a more  or  less  even  distribution  of  errors  across  all 
tasks  (assuming  the  equipment  interface  is  adequate  and  the  task  is  feasible 
with  the  equipment  provided).  By  comparing  measured  performance  with 
performance  standards  (if  they  exist)  and  with  the  material  covered  in  the 
participants'  training  course,  training  deficiencies  can  be  determined.  This 
information,  along  with  the  test  participants'  evaluation  of  their  own 
training,  provides  inputs  to  determine  training  and  personnel  selection 
adequacy.  Procedures  for  evaluating  the  adequacy  of  the  training  and  selec- 
tion are  well  documented  (29,  51,  64,  69,  71). 
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Problem  and  Error  Description.  Within  the  military  R&D  cormiunity,  there 
is  a growing  awareness  of  the  relationship  between  the  efficiency  with  which 
soldiers  are  able  to  operate  and  maintain  equipment  and  the  ultimate  effective- 
ness of  the  man-materiel  system  (77). 

"People  are  the  only  responsible  agents  in  the  system.  No  matter  how 
small  the  roles  assigned  to  people,  they  are  responsible  roles.  People 
determine  whether  the  system  is  ready  to  operate,  what  it  is  to  do,  how 
and  when  it  is  to  do  it,  when  and  what  variations  in  performance  are  to 
occur,  and  what  constitutes  adequate  or  complete  performance.  People 
decide,  control,  guide,  change,  and  evaluate.  They  are  expected  to 
anticipate,  detect,  compensate  for,  and  explain  any  undesirable  varia- 
tions in  perforiTiance.  And  their  errors  assume  a significance 
commensurate  with  their  responsibilities"  (49). 

Consequently,  to  evaluate  system  performance,  one  must  have  a fundamental 
understanding  of  the  reliability  of  the  human  component.  The  beginning  of 
that  understanding  is  with  the  measurement  of  human  performance  errors. 

Errors  can  be  defined  as: 

1.  Performance  of  a required  action  incorrectly 

2.  Performance  of  a required  action  out  of  sequence 

3.  Failure  to  perform  a required  action 

4.  Failure  to  perform  a required  action  within  an  allotted  time  period 

5.  Performance  of  a non- required  action, 

DI-H-1334A  requires  that  frequencies,  consequences,  and  causes  of  each 
error  identified  in  the  HFE  test  be  completely  specified.  To  assess  the 
impact  of  the  effects  of  human  performance  error  on  system  performance,  the 
frequency  and  consequences  of  the  error  must  be  determined.  Error  rates  are 
obtained  by  deriving  the  following  ratio  for  each  error  identified: 

Error  rate  = # of  times  error  conmitted 

total  # of  times  task  performed 

Error  consequences  can  be  obtained  by  performing  a Failure  Mode  and  Effects 
Analysis  (FMEA),  as  described  by  several  writers  (8,  27,  33,  60).  Procedures 
for  identifying  error  causes  include: 

1.  Direct  observation  of  operator  performance 

2.  Analyses  of  films  or  video  recording  of  task  group  performance 

Procedures  for  identifying  the  causes  of  human  error  are  also  described 
elsewhere  (8,  11,  28).  Error  causes  and  alternative  solutions  and  actions  to 
eliminate  or  reduce  errors  should  be  fully  described. 
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Man-Machine  Incompatibilities.  There  are  three  general  types  of  incom- 
patibilities  which  the  test  manager  should  be  alert  to  detect.  Although  all 
may  spring  from  system  design,  each  is  manifested  in  a somewhat  different  way. 

Task  group  interference  occurs  when  an  individual,  performing  his  assigned 
tasks  correctly  and  in  their  proper  order,  interferes  with  one  or  more  other 
individuals  who  are  attempting  to  perform  their  own  tasks  in  the  system. 
Examples  include; 

1.  The  requirement  for  two  different  people  to  use  the  same  control  at 
the  same  time  but  for  different  purposes 

2.  The  requirement  for  one  person  to  have  performed  Task  A before 
another  person  can  perform  Task  B;  where  Task  B is  now  ready  to  be 
performed,  but  Task  A has  not  yet  been  completed. 

Equipment  incompatibilities  arise  when  two  (or  more)  equipment  subsystems 
(each  of  which  may  meet  relevant  human  engineering  design  criteria)  cannot  be 
used  together.  Common  examples  of  this  are  sound-attenuating  ear  muffs  and 
helmets  (which  cannot  be  worn  at  the  same  time)  and  vehicle  crewmen  helmets 
and  optical  sights  (where  the  thickness  of  the  helmet  prevents  the  gunner 
from  attaining  the  proper  eye  relief  on  the  browpad  of  the  sight).  A more 
sophisticated  example  was  detected  during  the  test  of  the  RPV  control  station. 
The  control -display  relationships  on  the  runway  and  glidescope  monitor  were 
found  to  be  the  reverse  of  those  same  relationships  on  the  onboard  video  of 
the  RPV;  therefore,  performance  of  a control  input  while  observing  one 
display  would  be  likely  to  induce  errors  in  control  inputs  made  from  obser- 
vations of  the  other  display. 

The  third  general  type  of  man-machine  incompatibility  is  the  classic 
human  engineering  problem  of  improper  hardware  design  and  its  effect  on  the 
human  operator.  The  scope  of  this  category  is  broad  and  includes  not  only 
simple  problems  (such  as  equipment  which  is  too  heavy  or  controls  which 
cannot  be  reached  by  the  operator),  but  also  equipment  characteristics 
(e.g.,  vibration)  which  affect  human  performance  and  allocation  of  functions 
to  human  performance  (such  as  software  design  which  requires  the  operator  to 
retain  and  recall  more  information  than  can  be  dealt  with  effectively). 

Because  the  conduct  of  an  HFE  test  in  accordance  with  DI-H-1334A  requires 
actual  human  performance  of  tasks  on  prototype  hardware  (or  mock-up),  an 
excellent  opportunity  is  afforded  for  identifying  safety  hazards  in  operations 
and  maintenance.  Although  a preliminary  hazard  analysis  of  the  system  will 
normally  have  been  performed  prior  to  the  conduct  of  an  HFE  test  (60),  it  is 
difficult  in  a paper  study  to  anticipate  all  of  the  crew  and  operator 
equipment  interactions  which  can  create  hazards.  The  HF  test  manager  should 
review  the  preliminary  hazard  analysis  prior  to  developing  the  test  segments 
to  insure  those  hazards  are  addressed  (see  Test  Segment  Development  above). 
However,  during  the  actual  conduct  of  the  test,  the  test  manager  should  be 
alert  to  identify  any  other  safety  problems  which  may  appear.  Each  hazard 
identified  during  testing  should  be  described  in  paragraph  7 of  the  DI-H-1334A 
report.  The  frequency  with  which  each  hazard  was  encountered  by  the  test 
participants  should  be  stated  and,  if  this  frequency  is  not  reflected  in  the 
data,  the  discrepancy  should  be  explained.  The  possible  consequences  of  each 
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hazard  should  be  estimated  whenever  possible,  with  respect  not  only  to 
personnel  who  may  be  affected  but  also  with  respect  to  system  effectiveness 
and  reliability.  Where  a hazard  is  identified  prior  to  the  HFE  test,  the 
DI-H-1334A  report  should  state  whether  the  procedures  proposed  to  eliminate 
or  minimize  it  were  adequate.  Where  a hazard  is  identified  as  the  result  of 
the  HFE  test,  the  DI-H-1334A  report  should  recommend  alternatives  for 
avoiding  it.  Several  authors  have  described  procedures  for  identifying  and 
eliminating  safety  hazards  (3,  51,  60). 

DI-H-1334A  requires  that  alternative  solutions  be  proposed  for  all 
observed  problems,  errors,  incompatibilities  and  hazards.  Each  proposed 
solution  should  be  stated  in  terms  of  one  or  more  of  four  possibilities: 

1.  Equipment  redesign 

2.  Alteration  of  human  performance  tasks 

3.  Personnel  selection  criteria 

4.  Training 

The  alternative  solutions  proposed  should  reflect  both  the  stage  of  system 
development  and  the  HF  engineer's  appreciation  of  system  life  cycle  cost. 

For  example,  if  a problem  amenable  to  a hardware  fix  was  noted  during  the 
testing  of  an  advanced  development  prototype,  it  would  almost  certainly  be 
more  cost-effective  to  make  design  changes  during  engineering  development 
than  to  upgrade  the  eventual  operator's  selection  criteria  or  extend  his 
training.  Hardware  fixes  involve  non-recurring  costs.  Upgrading  the  QQPRI 
incurs  costs  which  must  be  lived  with  for  the  life  of  the  system. 

Since  the  final  choice  of  problem  solution  will  be  made  by  someone  else, 
the  HF  engineer  should  provide  enough  detail  in  each  of  his  alternative 
solutions  (e.g.,  does  a design  solution  refer  only  to  one  manufactured  part 
or  to  an  entire  component)  and  the  HF  engineer's  subsequent  prediction  of 
the  changes  in  system  performance  which  would  accrue  from  adoption  of  each 
alternative  proposed.  The  HF  engineer  is  not  expected  to  perform  a Cost  and 
Operational  Effectiveness  Analysis  (COEA).  The  engineer  should  realize, 
however,  that  this  section  of  the  DI-H-1334A  report  may  subsequently  be  used 
in  such  an  analysis. 

Human  Performance  Impact  Assessment.  As  Howard  and  Lipsett  have 
reported. 

Human-initiated  malfunctions. . .account  for  50  to  70  percent  of 
all  failures  of  major  weapons  and  space  systems...  This  places 
human  error  ahead  of  design  error,  component  unreliability,  and 
lapses  of  quality  control  in  manufacturing— that  is,  ahead  of 
electrical,  mechanical,  and  structural  failures  as  a source  of 
system  troubles;  As  a result,  a reliability  program  which  only 
addresses  hardware  spends  80  to  90  percent  of  its  budgeted 
engineering  cost  and  manpower  to  solve  less  than  50  percent  of 
the  total  reliability  problem  (15). 
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One  of  the  major  reasons  for  conducting  a 0I-H-1334A  evaluation  is  to 
determine  the  adequacy  with  which  system  personnel  can  perform  assigned 
duties.  The  determination  of  the  time  required  and  the  errors  committed  in 
performing  assigned  tasks  is  used  to  determine  the  impact  of  human  performance 
on  system  effectiveness  and  reliability.  Ideally,  the  impact  of  human  per- 
formance is  assessed  by  comparing  measured  task  performance  with  required 
task  performance  standards.  Various  quantitative  techniques  have  been 
developed  to  assess  the  impact  of  human  performance  in  system  reliability 
and  effectiveness  (8,  10,  25,  26,  27,  30,  32,  38,  43,  44,  48,  50).  Basically, 
most  of  these  techniques  consist  of  conducting  the  following  steps  (30): 

1.  Perform  a task  analysis  to  determine  the  behavioral  units  (e.g., 
steps,  subtasks,  tasks)  to  which  the  prediction  will  be  applied. 

2.  Analyze  the  behavioral  units  to  determine  which  parameters  must  be 
considered  in  making  the  prediction.  Typical  parameters  known  to 
affect  performance  that  might  be  considered  are:  organization  of 
controls/displays,  presence  or  absence  of  feedback  information, 
accuracy  required,  environmental  factors,  etc.  This  step  is  oHen 
performed  implicitly  and  only  the  most  obvious  parameters  might  be 
considered. 

3.  Assign  the  input  data  to  the  behavioral  units  whose  performance  is 
to  be  predicted.  This  involves  the  following  steps: 

a.  Determine  the  sources  available  for  supplying  the  input  data 

b.  Match  data  parameters  with  those  describing  the  units  to  be 
predicted  (e.g.,  if  the  data  have  been  collected  under  different 
conditions  than  those  in  the  task  being  assessed,  appropriate 
adjustments  have  to  be  made) 

c.  Assign  selected  data  to  predictive  units 

4.  Use  a selected  predictive  method  to  obtain  the  desired  output 
measures.  This  is  achieved  either  by  using  probability  statistics 
to  combine  behavioral  units  to  take  into  account  their  inter- 
dependencies, or  by  employing  simulation  techniques  to  achieve  the 
same  end. 

5.  Finally,  combine  the  predicted  human  reliability  estimate  with  the 
corresponding  hardware  estimate  to  give  an  overall  system  reliability 
figure. 

One  of  the  measures  from  which  an  emerging  system's  ultimate  effectiveness 
is  normally  predicted  is  its  operational  availability  (Aq).  Although  there 
are  a number  of  formulas  for  deriving  Aq,  most  employ  a term  identified  as 
Mean  Time  To  Repair  (MTTR)  (15).  This  value  may  be  calculated  for  the  purpose 
of  developmental  testing  from  the  DI-H-1334A  data  concerning  performance  times 
of  maintenance  tasks.  However,  the  full  contribution  of  human  performance  to 
Aq  will  not  have  been  assessed  unless  the  actual  number  of  human  errors  (by 
category)  measured  during  the  DI-H-1334A  test  is  compared  with  the 


43 


opportunities  for  error  (27),  and  that  ratio  used  either  in  the  calculation 
of  MTBF  (Mean  Time  Between  Failures)  or  whatever  other  term  is  used  to 
determine  the  measured  system  performance  reliability  component  of  the  Aq 
equation. 

For  many  tested  systems,  human  performance  standards  will  not  be 
available.  In  these  cases,  the  impact  of  the  operator's  tested  performance 
can  be  assessed  by  using  assumed  performance  standards  or  by  identifying  a 
baseline  task  within  the  set  of  tested  tasks  which  can  provide  a basis  of 
comparison.  Assumed  performance  standards  can  be  generated  on  the  basis  of 
performance  standards  from  similar  systems  or  on  the  basis  of  tested 
performance  of  other  task  groups  in  the  same  system.  When  neither  system 
performance  specifications  nor  human  performance  specifications  can  be 
identified,  then  a baseline  task  technique  can  be  used  to  provide  a form  of 
internal  validity.  In  this  technique,  all  obtained  performance  times  and 
error  rates  are  compared  with  performance  on  the  task  selected  as  a baseline. 
A task  is  selected  as  a baseline  if  it  is  (1)  pervasive  or  representative  of 
a wide  range  of  operator  duties,  (2)  performed  with  relatively  few  errors, 
and  (3)  performed  with  relatively  small  variations  in  performance  times 
among  test  participants.  This  procedure  is  illustrated  in  the  CCD  sample 
test . 


Differences  between  obtained  task  performance  and  either  assumed  standards 
or  a baseline  task  should  be  described  as  in  the  case  with  stated  performance 
standards.  However,  the  same  statistical  tests  may  not  be  appropriate  for 
discussing  the  reliability  of  the  observed  differences.  In  this  case,  the 
description  may  be  given  in  terms  of  the  qualitative  impact  of  the  tested 
operator  performance  on  projected  system  performance  (10,  30). 

After  the  differences  between  obtained  and  required  performances  have 
been  summarized,  those  obtained  performances  which  are  widely  divergent  from 
the  standards  should  be  analyzed  to  determine  the  reasons  for  the  differences. 
This  analysis  will  often  be  qualitative  rather  than  quantitative,  since  the 
source  of  the  differences  may  be  obscure.  In  those  instances  where  the 
performance  is  significantly  poorer  than  the  standard,  the  previous  dis- 
cussions of  problems  and  errors  may  already  have  provided  suggested  causes 
and  remedies  for  the  poor  performance.  In  the  case  of  tested  performance 
which  is  significantly  better  than  the  standard,  the  analysis  should  identify 
the  sources  of  the  superior  performance  and  suggest  whether  the  standard 
should  be  revised  to  account  for  these  identified  sources. 


Report  Preparation 

The  outline  contained  in  Block  10  of  DI-H-1334A  provides  a logical 
organization  of  all  the  required  test  items;  therefore,  this  outline  should 
be  used  to  draft  the  test  report.  The  two  sample  HFE  tests  are  organized 
according  to  this  outline  and  illustrate  the  format  and  amount  of  detail 
required.  These  sample  reports  can  be  used  to  resolve  ambiguities  which  may 
arise  while  drafting  a test  report. 
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The  first  section  of  the  test  report  is  intended  as  an  administrative 
introduction.  This  section  includes  a description  of  the  personnel  positions 
that  were  tested  and  a brief  summary  description  of  their  work  space, 
equipment,  and  major  performance  requirements.  This  section  also  includes 
the  details  of  test  administration,  including  test  location  and  the  personnel 
involved  in  conducting  and  supervising  the  test. 

Section  2 deals  with  planning  and  conducting  the  HFE  test.  The 
first  two  paragraphs  of  this  section  include  a detailed  taxonon^y  of  all 
tasks  performed  by  each  personnel  position  and  the  standards  to  which  the 
tasks  should  be  performed.  The  remaining  paragraphs  of  Section  2 are  devoted 
to  descriptions  of  the  test  environment,  participants,  and  procedures. 
Sufficient  detail  should  be  included  to  permit  the  reader  to  determine  the 
degree  to  which  the  test  conditions  match  the  expected  use  conditions  of  the 
tested  system.  Those  test  conditions  which  differ  from  the  expected  use 
conditions  are  described  in  the  final  paragraph  of  the  section. 

Section  3 of  the  report  describes  the  test  design,  methods,  procedures 
and  equipment  used  to  collect  the  data.  Where  tradeoffs  have  been  made  in 
selecting  among  alternative  designs  and  methods,  the  rationale  for  those 
selected  should  be  stated. 

Sections  4,  5,  6 and  7 give  the  results  and  describe  the  human  perfor- 
mance errors  and  safety  hazards  observed  during  testing.  Sufficient  detail 
should  be  included  in  these  sections  so  that  test  participants'  performance 
times  and  errors--as  well  as  their  subjective  responses— can  be  independently 
analyzed.  However,  extensive  tables  of  raw  data  should  be  avoided  except 
where  essential  to  illustrate  a problem  or  error. 

Section  5 also  describes  the  problems  observed.  In  this  context,  a 
problem  may  be  defined  as  the  cause  of  an  error.  Even  though  an  error  may 
not  have  been  observed  during  an  HFE  test,  analysis  may  show  that  it  will 
probably  occur  later.  For  example,  during  the  CCD  test,  observers  noted  that 
the  labels  were  wearing  off  the  rubberized  keys.  It  can  reasonably  be  assumed 
that,  when  the  labels  become  obscured,  errors  will  occur.  Such  problems, 
together  with  their  anticipated  consequences  and  recommended  solutions  or 
corrections,  should  be  described  in  the  test  report. 

Section  8 of  the  report  discusses  the  impact  of  the  observed  human 
performance  on  the  reliability,  availability  and  effectiveness  of  the  tested 
system.  The  first  subparagraph  of  this  section  calls  for  a statement  of 
system  performance  goals  to  provide  the  standards  by  which  the  impact  of 
observed  performance  can  be  assessed.  Notice  that  these  are  overall  system 
goals  rather  than  the  human  performance  goals  that  were  identified  previously 
in  Section  2.  Quantitative  analyses  and  calculations  should  be  included  as 
much  as  possible  to  aid  systems  analysis  personnel  in  combining  the  HFE  test 
results  in  their  calculations  of  overall  system  effectiveness. 

Section  9 sunmarizes  the  HFE  test's  conclusions  and  recommendations. 

This  sumnary  highlights  the  test's  primary  findings,  their  estimated  impacts 
on  system  performance,  and  recommended  changes.  It  is  especially  important 
that  this  summary  provide  a comprehensive,  clear,  and  concise  description 
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of  the  results  and  recommendations,  because  other  reports  may  use  it  as  a 
summary  of  the  HFE  test's  results. 
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INTRODUCTION 


Task  Group  Identification 

This  report  describes  the  procedures  and  results  of  a Human  Factors 
Engineering  (HFE)  test  of  the  Pilot  and  Copilot  task  groups  of  the  proposed 
Compass  COPE  Remotely  Piloted  Vehicle  (RPV)  system.  The  Pilot  and  Copilot 
are  responsible  for  tracking,  monitoring,  and  controlling  single  or  multiple 
remote  vehicles  during  takeoff,  en  route,  on-station,  and  landing  phases  of 
flight  missions. 


Task  Group  Summary 

The  Pilot  and  Copilot  operate  the  equipment  of  the  Ground  Control 
Station  (GCS),  which  is  housed  in  a standard  equipment  shelter  measuring 
approximately  8 x 8 x 20  feet.  The  GCS  van  contains  the  necessary  equipment 
for: 

1.  Tracking,  monitoring,  and  controlling  multiple  RPV's 

2.  Up-link  transmission  and  down-link  reception 

3.  Computer  and  data  processing 

4.  Voice  communication 

5.  Air  conditioning  and  lighting 

6.  GCS  system  testing. 

Each  operator  has  a control  console.  The  Pilot's  console  contains  the 
controls  and  displays  required  for  active  manual  control  of  a single  RPV. 
Additional  RPV’s  can  be  manually  controlled  from  this  station  by  sequentially 
selecting  each  vehicle  for  control.  In  addition,  the  Pilot's  console 
contains  a computer-driven  Multi-Function  Display  (MFD)  which  can  display 
single  or  multiple  vehicle  parameters.  The  Copilot/Flight  Engineer's 
console  includes  the  controls  and  displays  required  for  multiple  vehicle 
status  monitoring  and  for  single  vehicle  subsystems  monitoring.  This 
console  also  includes  controls  for  redundant  subsystems  selection,  as  well 
as  an  MFD  for  selectable  data  display. 


Test  Site 

The  HFE  test  was  conducted  at  Teledyne  Ryan  Ae'ronautical , San  Diego, 
California,  between  February  9 and  February  13,  1976.  Test  sessions  were 
conducted  daily  between  0830  and  1630.  The  tests  were  supervised  by  Mr. 
Barry  L.  Berson  and  Ms.  Marlene  Artof,  of  Perceptronics,  Inc. 


48 


HFE  TEST  PREPARATIONS 


Pilot  and  Copilot's  Task  Groups 

Operational  Performance  Requirements.  Tables  9 and  10  show  sequential 
lists  of  Pilot  and  Copilot  tasks  and  subtasks  for  all  phases  of  a mission. 

The  Pilot  serves  as  the  overall  mission  commander  and  performs  the  tasks 
required  for  manual  control  and  automatic  systems  monitoring  in  single  RPV 
operations.  These  single  vehicle  tasks  are  particularly  prevalent  during 
takeoff  and  landing  operations  and  during  emergency  or  non-normal  en  route 
operations. 

The  Copilot/Flight  Engineer  performs  somewhat  differently  from  a Copilot 
on  board  a manned  aircraft.  In  the  RPV  system,  the  Copilot's  tasks  include 
monitoring  and  systems  management  for  all  vehicles  under  GCS  control.  This 
monitoring  function  applies  particularly  to  the  several  RPV’s  that  are  flying 
under  fully  automatic  control  while  the  Pilot  attends  to  a single  vehicle 
under  partial  manual  control.  The  Copilot/Flight  Engineer's  tasks  also 
include  calculating  takeoff  and  landing  parameters,  attending  to  navigational 
problems,  communicating  with  various  radio-linked  groups  for  flight  clear- 
ances and  advisories,  and  entering  flight  plan  changes  into  the  computer. 

Maintenance  Performance  Requirements.  The  crew  members  (Pilot  and 
Copilot)  are  not  expected  to  perform  any  maintenance  functions  other  than 
adjusting  the  front  panel  controls.  The  preliminary  maintenance  concept  for 
the  Compass  COPE  system  is  to  have  an  on-line  backup  van  ready  to  assume 
control  of  the  RPV's  in  case  major  equipment  components  in  th»^  primary  van 
malfunction.  In  the  event  of  a catastrophic  GCS  equipment  failure,  the  crew 
members  would  transfer  to  the  backup  control  van  while  a general  maintenance 
team  repaired  the  primary  van.  For  a non-catastrophic  but  annoying  failure, 
a maintenance  technician  would  repair  the  device  during  a lull  in  the  mission 
period. 


Human  Performance  Standards 

The  Compass  COPE  system  requires  simultaneous  remote  control  of  up  to 
five  flight  vehicles.  The  design  of  the  system  requires  the  controllers  to 
monitor  the  automatic  operation  of  the  vehicles  and  to  control  manually  one 
of  the  vehicles  during  selected  routine  mission  phases  and  during  emergencies. 
Therefore,  the  operators  must  have  sufficient  information  and  control  cap- 
ability to  decide  when  to  assume  manual  control,  to  control  a vehicle  accur- 
ately, and  to  monitor  other  vehicles  under  automatic  control.  However, 
specific  human  performance  standards  for  the  task  groups  have  not  yet  been 
developed. 
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TABLE  9 


TASK  1 


TASK  2: 


TASK  3; 


Pilot's  Tasks  and  Subtasks 


PERFORH  PRE-TAXI  OPERATIONS 
Subtasks 

1.1  Read  (out  loud)  Subsystem  Parameters  to  copilot 

a)  Propulsion 

b)  Electrical 

c)  Flight  Control  System  (FCS) 

d)  Hydraulic 

e)  Environment 

f)  Prime  Mission  Equipment  (PM£) 

1.2  Turn  on  Heather  Radar  (HR) 

1.3  Verify  the  Weather  Radar  is  operational 

1.4  Command  Crew  Chief  (CC)  to  start  engine 

1.5  Verify  vehicle  on  internal  power 

1.6  Call  out  radar  range  to  Radar  Technician  (RT) 

1.7  Set  Federal  Aviation  Administration  (FAA) 
Transponder  to  correct  frequency 

1.8  Set  FAA  UHF  relay  link 

1.9  Set  brakes  on 

1.10  Connand  CC  to  install  safety  pins 

1.11  Monitor  Subsystems 

1.12  Connand  CC  to  pull  umbilical 

1.13  Read  (out  loud)  subsystem  parameters  to  copilot 

a)  Propulsion 

b)  Electrical 

c)  FCS 

d)  Hydraulic 

e)  Environment 

f)  PME 

1.14  Zero  all  connand  controls 

1.15  Verify  redundant  systems  are  operational 

1.16  Turn  TV  system  on 

1.17  Verify  video  on  wideband  data 

1.18  Adjust  zoom 

1.19  Adjust  CRT  if  required 

1.20  Verify  brakes  and  gear  operational 

TAXI  RPV  TO  RUNWAY 
Subtasks 

2.1  Increase  RPM 

2.2  Release  brakes 

2.3  Monitor  TV 

2.4  Maneuver  Joystick  for  taxi  on  runway 

2.5  Set  brake 

2.6  Continue  to  taxi,  steering  as  required 

2.7  Comnunicate  with  CC  to  check  if  wings  are  level 

2.8  Proceed  to  runway 

a)  Set  brakes 

b)  Increase  RPM 

c)  Steer  as  required 

2.9  Verify  RPV  on  track  and  steer  to  runway  heading 

2.10  Set  RPM  to  idle 

2.11  Set  Automatic  Flight  Control  System  (AFCS)  control 
to  takeoff 

2.12  Verify  with  RT  that  radar  altimeter  is  on 

2.13  Verify  with  CC  that  RPV  is  ready 

r PERFORM  TAKEOFF  PROCEDURES 
Subtasks 

3.1  Increase  RPM 

3.2  Read  (out  loud)  subsystem  parameters  to  copilot 


TASK  3:  PERFORM  TAKEOFF  PROCEDURES  (continued) 

3.6  Set  Electronic  Altitude  and  Direction  Indicator 
(EADI)  controls  for  TakeOff  (TO) 

3.7  Choose  Air  Traffic  Control  (ATC)  countdown 

3.8  Set  clock 

3.9  Set  intermediate  altitude 

3.10  Switch  brakes  off 

3.11  Set  RPM  for  takeoff 

3.12  Monitor  video  and  steer  as  required 

3.13  Verify  airspeed  and  runway  distance  is 
exceeding  abort  criteria 

3.14  Verify  rotation  and  initial  climbout  on  EADI 

3.15  Monitor  altitude 

3.16  Turn  to  climbout  heading 

3.17  Reset  airspeed  trim  to  $ 

3.18  Adjust  video  as  required 

3.19  Hold  altitude  as  required  to  dear 
aircraft  (AC) 

3.20  Test  left  and  right  turns 

3.21  Check  left  and  right  slue  capability 

3.22  Check  RPM 

3.23  Connand  landing  gear  up 

3.24  Monitor  vertical  and  airspeed  trim  changes 

3.25  Monitor  FCS  data 

3.26  Check  subsystem  parameters  for  normalcy 

TASK  4;  ASSUME  CONTROL  OF  RPV  FROM  THE  RUNWAY  CONTROL 
FACILITY 

Subtasks 

4.1  Verify  subsystem  status  and  significant 
events  with  RCF 

4.2  Countdown  for  handoff 

4.3  Assume  control  of  the  RPV 

4.4  Verification  of  handoff 

4.5  Verify  control  of  each  RPV 

TASK  5:  CONTROL  AND  MONITOR  RPV'S  DURING  EN  ROUI'E  PHASE  OF 
THE  MISSION 
Subtasks 

5.1  Turn  TV  system  off 

5.2  Turn  on  HR 

5.3  Monitor  HR  display 

5.4  Continually  monitor  subsystem  parameters 

5.5  Assume  manual  control  of  an  RPV  to  alter 
headings  or  to  remedy  equipment  malfunctions 

5.6  Verify  with  RT  that  all  RPV'S  are  on  appropriate 
track 

'ASK  6:  CONTROL  AND  MONITOR  RPV'S  DURING  THE  MISSION  PHASE 
Subtasks 

6.1  Turn  on  PME  at  the  initiation  point 

6.2  Verify  with  Mission  Control  (MC)  that  PME  is 
on  and  acquiring  desired  data 

6.3  Adjust  prograimned  flight  plans  of  RPV  if 
required 

6.4  Continually  monitor  RPV  subsystems  during  the 
mission 

6.5  Turn  PME  off  when  mission  is  completed 
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(continued) 


TASK  7:  CONTROL  AND  MONITOR  RPV'S  DURING  THE  RETURN 
PHASE  OF  THE  MISSION 

Subtasks 

7.1  Turn  off  mission  program  flight  plan 

7.2  Set  RPV  to  cruise  mode 

7.3  Check  control  of  each  RPV 

7.4  Continually  monitor  subsystem  parameters 

7.5  Assume  manual  control  of  an  RPV  If  required 

7.6  Verify  all  RPVs  are  on  track 

TASK  8:  LAND  RPVs 
Subtasks 

8.1  Lower  landing  gear 

8.2  Verify  landing  gear  Is  on  and  locked 

8.3  Extend  spoilers 

8.4  Monitor  environmental  Indicators 

8.5  Monitor  PCS  display 

8.6  Select  glide  slope 

8.7  Get  turnaround  If  required 

8.8  CoRinand  landing  node 

8.9  Turn  on  video 

8.10  Adjust  zoom 

8.11  Adjust  CRT 

6.12  Lower  flaps 

8.13  Verify  spoilers  are  active 

8.14  Set  and  monitor  flap 

8.15  Monitor  subsystem  parameters 

8.16  Check  fuel  remaining 

8.17  Comnand  landing  mode  on 

8.18  Acquire  RPV  track 

8.19  Command  couple/glide  slope 

8.20  Set  up  EAOI  data  presentation 

8.21  Set  automatic  landing  on 

8.22  Commimlcate  with  CC  about  landing  prediction 

8.23  Switch  landing  lights  on 

8.24  Switch  brake  off 

8.25  Align  RPV  with  runway 

8.26  Monitjr 

a)  MLS 

b)  PCS 

c)  Altitude 

d)  Video 


TASK  8:  LAND  RPVs  (continued) 

8.27  Adjust  video  zoom 

8.28  Set  airspeed  trim 

8.29  Adjust  RPM 

8.30  Set  vertical  trim 

8.31  Adjust  RPV  attitude 

8.32  Monitor  crab  angle  and  adjust  as  req>'ired 

8.33  Enable  nose  steering 

8.34  Monitor  video 

8.35  Monitor  MLS 

8.36  Steer  as  required 

8.37  Monitor  decrab 

8.38  Monitor  video 

8.39  Monitor  track  on  video  display 

8.40  Acquire  data  from  squat  switch 

8.41  Land  RPV 

8.42  Monitor  decrease  In  airspeed 

8.43  Monitor  braking 

8.44  Adjust  RPM  for  ground  Idle 

8.45  Switch  spoilers  to  full  out 
TASK  9:  STEER  RPVs  OPP  RUNWAY 

Subtasks 

9.1  Steer  RPV  manually  down  runway 

9.2  Set  brakes  on  at  specified  airspeed 

9.3  Monitor  airspeed 

9.4  Monitor  video 

9.5  Monitor  steering 

9.6  Set  brakes  off 

9.7  Adjust  RPM  as  required 

9.8  Steer  as  required 

9.9  Set  brakes  on  at  parking  spot 

9.10  Command  CC  to  Install  safety  pin 

9.11  Comnand  CC  to  turn  off  fuel 

9.12  Shutdown  vehicle  subsystems 

a)  Electrical 

b)  Engine 

c)  TV  video 

9. 13  Switch  flaps  up 

9.14  Switch  downlink  antenna  off 

9.15  Switch  landing  light  off 

9.16  Switch  radar  altimeter  off 

9.17  Turn  off  mission  recorder 
9.16  Pill  out  mission  reports 


(concluded) 
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TABLE  10 


Copilot's  Tasks  and  Subtasks 


TASK  1:  PRE-TAXI 
Subtasks 

1.1  Honitor  the  following  subsystem  parameters 

a)  Propulsion 

b)  Electrical 

c)  FCS 

d)  Hydraulic 

e)  Environment 

f)  PME 

1.2  Call  all  other  control  facilities;  verify 
on  line  and  available 

1.3  Verify  beacons  are  operational 

1.4  Set  FAA  relay  link 

1.5  Verify  flight-plan  clearance  w/Air  Traffic 
Control  (ATC) 

1.6  Obtain  range  clearance  from  ATC 

1.7  Notify  tower  of  status 

1.8  Notify  chase  aircraft  of  status 

1.9  Notify  MC  of  status 

1.10  Obtain  weather  clearance  data 

l.n  Perform  Microwave  Landing  System  (MLS)  self- 
test 

1.12  Monitor  subsystem  parameters 

1.13  Command  RT  to  adjust  down-link  antenna 

1.14  Verify  redundant  systems  are  operational 

1.15  Command  Rr  to  set  antenna 

TASK  2;  MONITOR  RPVs  DURING  TAXI  PHASE 
Subtasks 

2.1  Receive  clearance  to  taxi  fro*  tower 

2.2  Verify  MLS  on  and  coupled 

2.3  Cofrmand  RT  to  verify  antenna  selected 
TASK  3:  MONITOR  RPVs  DURING  TAKEOFF 

Sub tasks 

3.1  Obtain  ATC  clearance 

3.2  Monitor  subsystem  status 

3.3  Monitor  weather  displays 

3.4  Set  Identify  Friend/Foe  (IFF)  code 

3.5  Determine  abort  criteria  -time*  airspeed, 
distance 

3.6  Determine  flap  settings  and  call  out  to  pilot 

3.7  Determine  EAOI  settings  and  call  out  to  pilot 


3.8  CoRVnand  RT  to  set  down-link  antenna 

3.9  Notify  tower  of  takeoff 

3.10  Notify  all  stations  of  takeoff 

3.11  Monitor  distance/tum  up  airspeed  for 
aborted  TO 

3.12  Call  out  airspeed  to  pilot 

3.13  Honitor  all  subsystems 

3.14  Convnand  RT  to  check  antenna 

3.15  Verify  TO  okay 

3.16  Monitor  video  for  weather  conditions 

3.17  Call  out  climb  heading  to  pilot 

3.18  Verify  that  range  radar  and  beacon  track 
are  operational 

3.19  Verify  IFF  track  with  ATC 

3.20  Set  Ident/Flash  switch 

3.21  Verify  control  of  RPV  by  checking  with 
chase  AC 

3.22  Monitor  data  for  trim  changes 

3.23  Monitor  subsystem  paraireters 

TASK  4:  ASSUME  CONTROL  OF  RPV  FROM  THE  RUNWAY  CONTROL  FACILITY 
Subtasks 

4.1  Establish  conmunlcation  with  the  Runway  Control 
Facility  (RCF) 

4.2  Verify  MCF  #2  Is  on  line  and  available 

4.3  Command  RT  to  verify  down-link  coRiRunication 

4.4  Receive  report  of  subsystem  status  and  significant 
events  from  RCF 

4.5  Verify  plotting  on  CRT/plot  board 

4.6  Verify  control  of  RPVs  w/RCF,  MCF,  12.  ATC,  Range 

TASK  5:  MONITOR  RPVs  DURING  THE  EN  ROUTE  HHASE  OF  THE  MISSION 

Subtasks 

5.1  Turn  WR  on 

5.2  Set  WR  track  angle 

5.3  Set  WR  gain 

5.4  Set  WR  pulse 

5.5  Set  WR  sector 

5.6  Test  WR-on  "green" 

5.7  Adjust  the  WR  display 

5.8  Monitor  WR  data 

5.9  Set  up  for  programmed  pattern 

5.10  Verify  and  calculate  navigation  settings  for 
RPVs 

5.11  Correlate  RPV  positions  with  MC 


(continued) 
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TASK  6:  MISSION 


TASK  8:  MONITOR  RPVs  DURING  LANDING 


Subtasks 

6.1  Verify  that  the  PME  Is  on 

6.2  CoKinunlcate  with  MC 

6.3  Connand  RT  to  adjust  antenna 

6.4  At  Initiation  point.  Initiate  program  pattern 

6.5  Coinnunicate  with  MC 

6.6  Adjust  flight  plan  per  prime  MC  coinund 

6.7  Prepare  new  navigation  parameters 

6.8  Verify  and  Input  navigation  pattern  Into 
airborne  system 

6.9  Coordinate  with  ATC  for  revised  flight  track 
pattern 

6.10  Verify  new  pattern  with  prime  MC 

6.11  Correlate  with  MCF  #2,  RT,  and  pilot 

6.12  Verify  flight  plan  parameters 

6.13  Check  subsystems  status 

6.14  Obtain  weather  update  In  landing  area  from 
tower 

6.15  Verify  that  the  PME  Is  off 

6.16  Notify  ATC  RPVs  returning  to  base 

6.17  Monitor  electrical  system  data 

6.18  Turn  on  MR 

6.19  Adjust  WR  as  required 

TASK  7:  MONITOR  RPVs  DURING  THE  RETURN  PHASE  OF  THE  MISSION 
Subtasks 

7.1  Obtain  en  route  weather  data 

7.2  Obtain  predicted  landing  weather  data 

7.3  Notify  RCF  for  recovery 

7.4  Notify  MCF  12  of  return 

7.5  Advise  predicted  landing  time  to  ATC/tower 

7.6  Turn  program  pattern  off;  turn  cruise  mode  on 

7.7  Monitor  TV  for  traffic  and  weather  changes 

7.8  Report  position  and  altitude  to  ATC/tower 

7.9  Monitor  engine  and  FCS  subsystems 

7.10  Monitor  environment  subsystems 

7.11  Monitor  environment  displays 

7.12  Report  subsystem  status  and  significant  events 
to  RCF 

7.13  Verify  plot/tracing  position  of  MCF/RCF 


Subtasks 

8.1  Obtain  runway  selection  from  ATC 

8.2  Turn  MLS  on 

8.3  Perform  MLS  test 

8.4  Obtain  runway  clearance  data 

8.5  Monitor  environmental  data 

8.6  Determine  RPV  weight 

8.7  Command  RT  to  select  MLS  antenna 

8.8  Turn  off  HR 

8.9  Check  with  ATC  for  radar  vectors 

8.10  Advise  ATC  of  altitude  du  ' \g  descent 

8.11  Advise  tower  of  the  following 

a)  Request  for  approach 

b)  For  downwind 

c)  On  base 

d)  Turning  to  final 

8.12  Verify  MLS  is  operational 

8.13  Report  status  to  ATC/tower 

8.14  Verify  control  Is  accepted  by  MLS 

8.15  Establish  go  around  criteria 

8.16  Advise  tower  of  final  approach— distance 
and  altitude 

8.17  Monitor  MLS,  FCS,  and  altitude 

8.18  Verify  wind,  RPV,  weight,  and  temperature 
condition 

8.19  Advise  ATC  of  landing 
TASK  9:  PERFORM  RPV  POST-LANDING  DUTIES 

Subtasks 

9.1  Verify  MLS  Is  decoupled 

9.2  Monitor  and  call  out  airspeed 

9.3  Turn  off  MLS 

9.4  Request  ground  control  clearance  to  taxi 

9.5  Turn  off  HR 

9.6  Record  time  of  landing,  power  down 

9.7  Prepare  mission  reports 

9.8  Call  out  systems  to  be  shut  down:  electrical, 
engine,  TV  video,  etc. 

9.9  Advise  ATC,  tower,  and  MC,  "Mission  complete" 
9. ID  Conmunlcate  to  all  facilities  "Close  of  flight 


(concluded) 
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Test  Environment 


An  air  conditioned,  carpeted,  and  draped  conference  room  at  Teledyne 
Ryan  Aeronautical  was  used  for  testing.  The  room  was  located  in  a vacant 
section  of  one  of  their  buildings.  Components  from  the  TECOM  HFE  instrument 
package  were  used  to  record  the  following  environmental  conditions: 


Instrument 

p 

Spectra  Photometer 
Psychro-Dyne  Psychrometer 


Precision  Sound  Level  Meter 
and  Analyzer 

Each  measure  was  recorded  on  the  first  and  last  test  days.  Environmental 
measures  were  not  recorded  more  frequently  because  the  test  was  conducted  in 
an  unchanging  environment.  The  measures  were  replicated  during  the  last  day 
of  testing  to  verify  the  environment's  stability.  Table  11  presents  the 
mean,  standard  deviation  and  number  of  times  each  measure  was  recorded.  The 
small  standard  deviations  demonstrate  the  stability  of  the  environment. 


Measure 

Illumination 

Temperature  and 
Relative  Humidity 

Steady  State  Noise 


Test  Participants 

Three  Teledyne  Ryan  personnel  served  as  the  participants  for  the  HFE 
test.  Participant  characteristics  are  summarized  in  Table  12.  A GPM  an- 
thropological instrument  kit  was  used  to  record  participant  anthropometric 
data.  Since  operational  personnel  will  be  seated  during  mission  performance, 
detailed  measurements  for  seated  operators  were  obtained.  Two  tests  were 
administered  to  assess  the  participants'  intellectual  capabilities.  The  Otis- 
Lennon  Mental  Ability  Test  (Form  J)  was  administered  to  obtain  a measure  of 
the  participants'  general  verbal  and  mathematical  skills.  The  Bennett  Me- 
chanical Comprehension  Test  (Form  S)  was  administered  to  each  participant  to 
assess  their  mechanical  and  spatial  aptitudes.  Test  results  are  contained 
in  Table  12. 

The  participants'  professional  backgrounds  and  flight/RPV  experience 
are  listed  below: 


PARTICIPANT 

1 

2 

3 


PROFESSIONAL  POSITION 
Human  Factors  Engineer 
Systems  Engineer 

Systems  Engineer 


FLIGHT/RPV  EXPERIENCE 

Instrument  Instructor  Pilot 

Experienced  RPV  and  Air 
Traffic  Controller 

Experienced  RPV  Controller 
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TABLE  11 


GCS  Environmental  Recordings 


Average  Reading  Standard  Deviation  N 


Illumination  (Ft.  candles) 


a.  Console  Desk  Top 

156 

1.2 

2 

b.  Console  Front  Panel 

150 

.4 

2 

Temperature 

23°C 

2.8 

10 

Relative  Humidity 

45% 

2.4 

10 

Steady  State  Noise  (db  SPL) 
(Weighting) 


a. 

A 

45 

1.2 

2 

b. 

B 

46 

.58 

2 

c. 

C 

50 

1.3 

2 

d. 

Flat 

54 

.86 

2 
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TABLE  12 


Characteristics  of  Test  Participants 


PARTICIPANT 


VARIABLE 

1 

2 

3 

AGE  (years) 

43 

34 

52 

WEIGHT  (pounds) 

168 

155 

195 

FLIGHT  EXPERIENCE  (years) 

21 

0 

0 

RPV  EXPERIENCE  (years) 

9 (Engineering  5 
only;  no 
flight 
experience) 

10 

ATC  EXPERIENCE  (years) 

0 

0 

6 

SCHOOL  (last  year  completed) 

M.S. 

B.S. 

3rd -year 
College 

VISUAL  ACUITY 

20/20 

20/25 

20/20 

PHYSICAL  DISABILITIES 

None 

None 

None 

OTIS-LENNON  PERCENTILE  RANK* 

97% 

84% 

76% 

BENNETT  PERCENTILE  RANK* 

97% 

98% 

99% 

PARTICIPANT 

BODY  DIMENSIONS  (inches) 

1 

2 

3 

STANDING  HEIGHT 

69 

66 

73 

SITTING  HEIGHT 

49.6 

49.6 

50.8 

SITTING  EYE  HEIGHT 

45.4 

45.3 

47.5 

ELBOW  RESTING  HEIGHT 

26 

26.1 

25.9 

HORIZONTAL  ARM  REACH 

30.6 

28.1 

32.5 

VERTICAL  ARM  REACH 

51.5 

48.4 

52 

ELBOW-FINGERTIP  REACH 

18.4 

17.6 

20.2 

BUTTOCK- HEEL  LENGTH 

44.8 

40.5 

46.3 

♦Twelfth-Grade  High  School  students  used  as  the  reference  group 
(Otis  Lennon,  N=1500;  Bennett,  N=2350) 


Test  Participants'  Clothing  and  Personal  b^uipment 

Compass  COPE  missions  will  be  directed  from  environmentally  controlled 
vans.  Operational  personnel  are  expected  to  be  dressed  in  working  uniforms 
without  protective  masks.  The  test  participants  were  dressed  in  their  normal 
working  attire  (shirt,  tie,  and  slacks),  which  is  comparable  to  uniforms  worn 
by  military  personnel  performing  desk-type  duties. 


Participant  Training 

The  participants  were  selected  on  the  basis  of  their  previous  experience. 
All  were  familiar  with  aircraft  and  RPV  control.  Each  of  the  operators 
received  two  hours  of  training  on  the  day  prior  to  the  first  test  day.  During 
this  two-hour  period,  the  operators: 

1.  Became  familiar  with  the  configuration  of  the  mock-up 

2.  Received  instructions  on  how  the  test  was  to  be  conducted 

3.  Practiced  the  test  segment  tasks  for  1-1/2  hours. 

Before  each  test  segment  began,  the  operators  were  briefed.  Each  task  listed 
in  the  scenario,  and  operator  response  requirements  for  those  tasks,  was 
discussed  prior  to  testing. 


Test  Equipment 

A full  scale  static  mock-up  of  the  GCS  console  (Figure  3)  was  used  to 
conduct  the  HFE  test.  The  mock-up  was  made  of  plywood  and  sheet  metal. 

Strips  of  magnetic  material  held  the  simulated  equipment  components  in  place. 
The  push  buttons  were  color  coded  to  indicate  control  conditions.  Four  sets 
of  buttons  were  used  to  change  control  states.  The  following  color  coding 
scheme  was  used: 


ON  = blue  CAUTION  = yellow 

OFF  = white  DANGER  = red 

The  dimensions  of  the  control  console  are  given  in  Figure  4.  Figure  5 shows 

the  details  of  the  mock-up's  front  panel. 


Deviations  of  Test  From  Expected  Use  Conditions 

1.  Static  controls  and  displays  were  used  during  testing 

2.  Highly  experienced,  single  drone,  RPV  controllers  were  used 
as  test  participants 

3.  Test  mission  times  were  truncated  (operational  mission  may  last 
up  to  thirty  hours;  test  missions  required  about  three  hours 

to  complete). 
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Figure  3.  Static  mock-up  of  the  Ground  Control  Station  (GCS) 
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BUTTON  STORAGE 


f 

Figure  5.  Details  of  GCS  mock-up  front  panel 
(Foldout) 
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Details  of  GCS  mock-up  front  panel 
(Foldout) 
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These  deviations  reflect  the  early  stage  of  design  for  the  RPV  control 
stations  where  dynamic  simulations  have  yet  to  be  developed  and  personnel 
have  yet  to  be  selected  or  trained.  The  focus  of  this  HFE  test  includes 
determining  (1)  the  feasibility  of  assigned  operator  tasks,  (2)  the  adequacy 
of  work  space  layout,  and  (3)  whether  all  required  displays  and  controls  are 
included.  The  deviations  did  not  affect  the  obtained  data. 


DATA  COLLECTION  TECHNIQUES 


Test  Plan 

The  test  design  consisted  of  testing  each  of  the  three  operators  in  all 
six  of  the  test  segments.  The  segments  included  performing  normal  and  con- 
tingency/emergency takeoffs,  missions,  and  landings.  In  the  normal  segments, 
no  equipment  malfunctions  were  simulated.  In  the  contingency/emergency 
segments,  equipment  malfunctions  and  flight  plan  changes  were  simulated  to 
assess  how  the  burden  of  these  occurrences  affects  operator  performance. 

Each  operator  was  tested  at  both  of  the  personnel  positions  for  each  of 
the  six  segments.  A total  of  six  sessions  was  conducted.  Two  operators  were 
evaluated  during  a session.  Each  session  consisted  of  three  segments.  The 
plan  for  this  study  is  shown  below.  The  numbers  listed  under  Pilot  and  Co- 
pilot indicate  test  operator  number.  Each  session  lasted  about  three  hours. 


TEST  SESSION 

TEST 

PILOT 

POSITION 

PARTICIPANT 

conuoT 

POSITION 

TEST  SEGMENTS 

1 

1 

2 

1,2,  and  3 

2 

3 

1 

1,2,  and  3 

3 

2 

3 

1,2,  and  3 

4 

2 

1 

. 4,  5,  and  6 

5 

1 

3 

4,  5,  and  6 

6 

3 

2 

4,  5,  and  6 

Data  Collection  Methods  and  Equipment 

Both  objective  performance  and  subjective  questionnaire  techniques  were 
used  to  obtain  the  information  necessary  for  evaluating  the  Pilot  and  Copilot 
task  groups.  The  data  collection  methods  used  included: 

1.  Behavioral  checklist 

2.  Operator  questionnaire 

3.  Human  factors  operations  checklist 
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4.  End  of  test  session  interview 

5.  Control  and  display  criticality  card  sort. 

The  primary  method  for  collecting  operator  performance  data  was  the 
behavioral  checklist.  Checklists  were  developed  for  each  task  group  and  test 
segment.  Each  checklist  contained: 

1.  A sequential  list  of  all  operator  tasks 

2.  Descriptions  of  the  required  operator  responses  for  each 
required  task 

3.  Columns  for  recording  task  response  duration,  response  adequacy 
(correct/error)  and  error  descriptions. 

On  the  checklist,  the  tasks  were  arranged  in  the  same  order  that  the 
operators  performed  them.  To  evaluate  operator  responses,  the  test  monitors 
would  observe  the  respective  operator's  response  and  compare  it  to  the 
response  required,  as  shown  on  the  checklist.  When  the  operator  performed 
the  task,  the  test  monitor  would  place  a check  (/)  in  the  appropriate  column 
of  the  checklist.  If  the  operatoi"  erred,  the  test  monitor  would  record  a 
brief  description  of  the  error  in  the  appropriate  column.  The  test  partici- 
pants' comments  about  task  sequencing,  adequacy  of  workspace  configuration, 
etc.,  were  also  recorded  on  the  checklist. 

The  procedures  for  collecting  performance  data  were  the  same  for  each 
of  the  six  test  segments.  Prior  to  testing,  the  test  participants  were 
given  scripts.  These  scripts  consisted  of  a detailed  description  of  Pilot 
and  Copilot  task  requirements.  Since  a high  percentage  of  operator  tasks 
involved  interaction  between  the  two  operators,  the  scripts  contained  des- 
criptions of  both  operators'  tasks.  These  scripts  allowed  the  operators  to 
coordinate  their  activities  by  telling  them  what  each  operator  was  required 
to  do  and  when  he  was  to  do  it.  To  insure  that  testing  would  run  smoothly, 
each  task  listed  on  the  scripts  was  fully  discussed  prior  to  testing. 

A test  monitor  observed  and  evaluated  the  performance  of  each  partici- 
pant. Each  test  monitor  used  a stopwatch  to  determine  the  task  response 
times.  Task  response  times  were  measured  from  the  time  the  test  director 
said,  "Start  task"  until  the  participant  said,  "Task  completed."  A black 
and  white  video  tape  recorder  was  used  to  record  problems  and  errors,  and 
to  obtain  documentary  footage.  A cassette  tape  recorder  was  used  to  obtain 
audio  recordings  of  the  operators'  communications. 


Data  Reduction  Methods 

The  data  from  this  test  of  a system  in  the  experimental  prototype  stage 
of  development  are  primarily  frequency  data,  thus  the  most  appropriate  method 
of  summarizing  such  data  was  to  calculate  the  means  and  standard  deviations. 
In  summarizing  the  questionnaires,  the  test  participants'  suggestions  and 
comments  are  arranged  logically,  but  reported  directly.  For  the  criticality 
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and  frequency  of  use  ratings  of  controls  and  displays,  rank  order  correlations 
and  coefficients  of  concordance  were  calculated,  with  a 5%  confidence  level 
used  in  both  cases. 


HFE  TEST  RESULTS 


This  section  presents  results  of  the  HFE  test  of  the  RPV  Pilot  and  Co- 
pilot personnel  positions.  It  summarizes  the  results  of  both  objective 
performance  and  subjective  questionnaire  data. 


Performance  Data 

Results  for  each  of  the  six  test  segments  are  summarized  below.  Table 
13  gives  the  mean  times  the  Pilot  and  Copilot  required  to  complete  each  task 
segment.  The  times  indicated  reflect  the  time  required  to  perform  the 
required  tasks.  On  many  tasks,  the  Copilot  completed  his  requirements  before 
the  Pilot  completed  his.  This  is  indicated  by  the  longer  segment  times  list- 
ed for  the  Pilot. 

Normal  Takeoff,  Mission,  and  Landing  Test  Segments.  No  equipment  mal- 
functions  or  requests  for  course  changes  were  simulated  in  the  first  three 
test  segments.  The  Pilot  and  Copilot  were  required  to  perform  all  of  the 
tasks  listed  in  Tables  9 and  10  for  these  three  segments.  In  the  normal 
takeoff  and  landing  segments,  the  operators  were  required  to  track,  monitor, 
and  control  a single  RPV.  In  the  normal  mission  segment,  they  were  required 
to  handle  three  RPV's. 

Simulated  Failures  in  Takeoff,  Mission,  and  Landing  Test  Segments.  I n 
test  segments  four  through  six,  the  operators  were  required  to  perform  all 
of  the  tasks  required  in  the  first  three  segments  plus  additional  tasks 
imposed  by  simulated  equipment  malfunctions  and  requests  to  alter  the  course 
of  several  RPV's.  The  operators  were  required  to  perform  takeoff,  mission, 
and  landing  operations  with  one,  three,  and  two  RPV's  respectively. 


Questionnaire  Data 

Operator  Questionnaire.  Participants  were  asked  to  fill  out  a question- 
naire wFrciTcovirid'Thi”?^!  owing  areas; 

1.  Biographic  data  (summarized  previously  in  Table  12) 

2.  Training  and  experience 

3.  RPV  operating  procedures 

4.  Task  interest 

5.  Man-machine  interface. 
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TABLE  13 


Task-Segment  Times  for  Pilot  and  Copilot 


PILOT 

COPILOT  ' 

TASK  SEGMENT 

MEAN 

SD 

MEAN 

1 

SD 

yi 

1 

21.3 

3.6 

16.4 

2.1 

2 

15.3 

1.4 

13.6 

1.5 

3 

11.5 

2.0 

9.0 

5.2 

4 

19.8 

5.5 

14.7 

4.0 

5 

21.1 

6.7 

22.2 

8.6  , ■ 

6 

18.5 

4.5 

15.6 

6.2 

TOTAL 

= 107.5 

Minutes 

91.5 

Minutes 

Times  for  each 

segment  were  recorded 

for  all  three  subjects. 
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The  participants  filled  out  the  biographical  section  of  the  questionnaire 
before  testing  and  completed  the  questionnaire  after  they  had  finished 
testing.  For  the  task  interest  and  man-machine  interface  sections  of  the 
questionnaire,  the  test  participants  were  told  to  imagine  themselves  in  the 
expected  operational  environment  and  to  answer  the  questions  accordingly. 

The  major  findings  are  summarized  in  the  following  paragraphs. 

Participant  Training  and  Experience.  The  experience  of  the  participants 
varied  widely.  Two  of  the  participants  were  experienced  RPV  operators,  one 
having  six  years  experience  in  RPV  range  control.  The  third  participant  had 
an  instrument  pilot  rating  and  twenty-one  years  of  flying  experience.  Col- 
lectively, the  participants  felt  that  some  previous  experience  in  fields 
such  as  systems  engineering,  air  traffic  control,  radio  communication,  and 
aviation  was  essential  for  RPV  Pilots  and  Copilots.  Table  14  shows  the 
amount  of  time  the  participants  felt  trainees  should  spend  on  types  of 
training,  and  training  for  areas  of  proficiency,  respectively. 

Task  Interest.  All  participants  found  the  task  of  operating  RPV's 
interesting,  the  participants  thought  that  the  mean  onset  time  for  boredom 
would  be  about  two  hours,  while  the  mean  onset  time  for  fatigue  would  be 
about  four  hours.  One  operator  suggested  that  length-of-duty  should  be 
dependent  on  mission  phase. 

RPV  Operating  Procedures.  Participants  felt  that  duties  in  such  areas 
as  handoffs  between  control  facilities  and  other  tasks  requiring  coordination 
should  be  further  defined  because  these  operations  are  so  complex.  Coord- 
ination procedures  will  be  studied,  and  recommendations  will  be  made  later 
in  system  development. 

Man-Machine  Interface.  The  participants  made  the  following  suggestions 
about  the  man-machine  interface: 

1.  Relocate  RPV  select  controls  from  desk  top  to  upright  panel 

2.  Relocate  go-around  switch  to  position  on  upright  panel  near 
throttle  control 

3.  Place  status  indicators  directly  above  EAOI  rather  than  to 
the  side 

4.  Move  video  monitor  controls  to  EADI  panel 

5.  Slant  Pilot  and  Copilot  panels  for  better  cross  correlation 
of  data  and  to  reduce  parallax  when  reading  other  position's 
data 

6.  Add  oil  quantity  or  low  oil  quantity  indicator 

7.  Add  MLS  indicator  for  operation/failure 

8.  Add  flap  position  indicator 

9.  Add  fuel  off  arm/fuel  off  command  controls. 
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TABLE  14 


Estimated  Training  Time  for  Each  Area  of  Proficiency 


RECOMMENDED  TRAINING 

TRAINING  AREA  TIME  (HOURS) 


MEAN 

SD 

Operation  of  RPV  System 

67 

18.4 

Checkout  of  RPV  System 

42 

10.2 

Takeoff 

10 

3.4 

Cruise  climb 

10 

2.8 

Descent 

10 

4.8 

Approach  and  landing 

78 

20.3 

Operation  of  GCS  TV  monitor 

16 

5.4 

Takeoff-abort 

8 

6.6 

Landing  wave-off 

16 

3.8 

Safety  and  emergency  procedures 

40 

9.6 

Flight  restrictions 

20 

5.3 

Radio-voice  communication  techniques, 
procedures,  etc. 

13 

4.6 

Backup  system  operation 

19 

6.3 

Multiple  RPV  operation 

100 

24.8 

N=3 
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10.  Add  takeoff-abort  switch, 

n.  Add  fuel  pressure  indicator. 

12.  Add  arm  rest  for  side  controller. 

13.  Add  glide-scope  command. 

14.  Add  glide-scope  display  video  monitor. 

15.  AFS  control  panel  should  have  an  indicator  for  RPV  under 

control . 

16.  Pattern  plotter  should  show  track  and  predicted  plot. 

17.  Need  destruct-armed  indicator. 

18.  Need  a "tie  in"  from  joystick  to  flight  program  indicator  and/ 
or  to  audio  alarm,  in  case  joystick  is  inadvertently  touched 
and  RPV  goes  off  programmed  flight. 

Criticisms  made  by  participants  included: 

1.  Main  Plotboard:  Distance  of  main  plotboard  from  Pilot  and 
Copilot  positions  makes  it  difficult  to  view  the  plotboard. 

2.  Command  Designate  Switch:  If  Pilot  does  not  designate  the 
appropriate  RPV,  Copilot's  command  goes  to  wrong  RPV. 

3.  Video  Monitor:  Current  display  format  leads  to  altitude  and 
orientation  errors. 

4.  Airspeed  and  Vertical  Trim:  No  indicator  showing  that  the  RPV 
accepts  the  value  set. 

All  participants  felt  that  their  operating  positions  were  comfortable 
and  that  there  was  no  difficulty  in  reaching  controls. 


Human  Factors  Operation  Checklist 

Each  participant  filled  out  a Human  Factors  Operation  Checklist.  The 
problems  that  were  reported  are  listed  below: 

1.  Not  all  operators  requiring  information  from  a common  display 
have  a clear  line  of  sight  from  their  operating  positions. 

2.  Not  all  primary  controls  and  displays  are  placed  within  the 
readily  accessible  visual  and  manual  spaces  on  the  console. 

3.  Emergency  controls  and  displays  are  not  placed  in  readily 
accessible  positions. 
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4.  In  some  cases,  the  operator's  hand  blocks  the  view  of  a display 
when  operating  the  associated  control. 

5.  Some  displays  cannot  be  read  easily  from  the  normal  location  of 
the  operator  requiring  the  information. 

6.  Controls  associated  with  the  right  hand  are  not  always  located 
below  or  to  the  right  of  their  displays  and  vice  versa  for 
controls  operated  by  the  left  hand. 

7.  Shapes  of  controls  cannot  always  be  discerned  by  touch. 


Criticality,  Frequency,  and  Sequence-of-Use  of  Controls  and  Displays 

Each  participant  ranked  all  of  the  Pilot's  and  Copilot's  controls  and 
displays  in  terms  of  criticality  (i.e.,  the  consequence  of  their  failure  on 
mission  success).  One  set  of  rankings  was  obtained  from  each  of  the  follow- 
ing four  categories: 

1 . Pilot  controls 

2.  Pilot  displays 

3.  Copilot  controls 

4.  Copilot  displays. 

Each  set  was  ranked  independently.  The  coefficient  of  concordance  (oi)  was 
calculated  to  determine  the  commonality  (level  of  agreement)  among  the  three 
participants.  Significant  agreement  was  obtained  for  each  category,  indicat- 
ing that  the  participants  generally  agreed  about  the  relative  criticality 
of  the  displays  and  controls.  Mean  rankings*  for  each  control  and  display, 
coefficients  of  concordance  and  levels  of  significance  (£),  for  each  of  the 
four  sets  are  shown  in  Tables  15  and  16. 

Work  process  charts  were  developed  to  determine  frequency  and  sequence 
of  use  of  all  mock-up  controls  and  displays.  Separate  charts  were  developed 
for  each  personnel  position  (Pilot  and  Copilot)  and  for  each  of  the  six  test 
segments.  Table  17  displays  a work  process  chart  for  the  Pilot  performing 
a normal  en  route  mission.  The  numbers  in  each  column  indicate  the  sequence 
of  instrument  use  for  each  task.  Information  for  the  charts  was  derived  from 
an  analysis  of  the  human  performance  requirements  and  from  an  evaluation  of 
operator  performance  during  testing. 

The  frequencies  of  Pilot  and  Copilot  use  of  controls  and  displays  are 
shown  in  Tables  15  and  16.  Controls  and  displays  are  listed  in  order  of 
criticality.  The  first  control/display  listed  was  judged  by  the  test  par- 
ticipants to  be  most  critical,  etc.  Frequency  data  was  also  ranked  to  provide 
a comparison  to  assessed  criticalities.  Rank  order  correlation  coefficients 
were  derived  to  evaluate  the  relationship  between  criticalities  and  frequency 
of  use  (i.e.,  were  the  components  that  were  judged  to  be  critical  also  used 
most  frequently?).  Significant  rank  order  correlation  coefficients  were 
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TABLE  15 


Rank  Orderings 

of  the 
Pilot 

Criticality  and  Frequency  of 
's  Controls  and  Displays 

Use  of  the 

MEAN 

CRITICALITY 

CRITICALITY 

FREQUENCY 

RANK 

RANK  ORDER 

RANK 

DISPLAY 

Airspeed  Indicator 

2 

1 

7 

Barometric  Altimeter 

2.3 

2 

4 

Radar  Altimeter 

4.7 

3 

4 

Vertical  Speed 

5.3 

4 

8 

EAOl 

6.7 

5.5 

6 

RPM 

6.7 

5.5 

2 

EGT 

7.0 

7 

3 

Spoiler 

7.3 

8 

9 

Distance  to  Go 

8.7 

9 

19 

Mode  Indicator 

11.3 

16.5 

15 

MLS 

11.3 

10.5 

15 

Flaps 

13.3 

12 

12 

Audio  Warning 

14.7 

13 

19 

Landing  Gear 

15.0 

14.5 

11 

Fuel  Gauge 

15.0 

14.5 

13 

Weather  Radar 

16.0 

16.5 

18 

ADI 

16.0 

16.5 

10 

MFD 

17.7 

18 

1 

Landing  Light 

19.3 

19 

13 

Clock 

19.7 

20 

15 

«.  • .77 

r • .56 

p < .001 

p < .01 

CONTROL 

RPV  Select 

2 

1 

2 

Command  Guidance 

3.7 

2 

9 

Throttle 

4 

3 

5 

Joystick 

6 

4 

2 

Brakes 

6.7 

5 

6 

AFCS 

7.3 

6 

7 

Landing  Mode 

8.3 

7 

12 

Spoiler 

8.7 

8 

n 

Flaps 

9.0 

9.5 

16 

Landing  Gear 

9.0 

9.5 

8 

MLS 

12.7 

11 

21 

Airspeed  Trim 

13 

12.5 

12 

Vertical  Speed  Trim 

13 

12.5 

12 

ADI 

15 

14 

21 

Oestruct 

15.7 

15 

24 

EADI 

16 

16 

10 

HSI 

16.7 

17 

21 

Radar  Altimeter 

17 

16 

18 

NAV  Mode 

18.3 

19 

12 

Audio  Warning 

19.7 

20 

25 

MFD 

20.7 

21 

4 

Intercom 

21.7 

22 

1 

Weather  Radar 

23 

23.5 

18 

Push  to  Talk 

23 

23.5 

25 

Landing  Light 

26 

26 

17 

Headset 

26 

26 

25 

Transponder 

26 

26 

20 

«•  • .83 

Pi  .001 


r » .S6 


P i .01 
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TABLE  16 


Rank  Orderings  of  the  Criticality  and  Frequency  of  Use  of  the 
Copilot's  Controls  and  Displays 


ntAN 

criticality 

CRITICALITY 

FREQUENCY 

RANK 

rank  order 

RANK 

display 

Electrical 

3.7 

1.5 

3 

Hydnulic 

3.7 

1.5 

1 

Environmental 

4.3 

3.5 

6 

Propulsion 

4.3 

3.5 

4 

FCS 

5.7 

5 

2 

MFD 

6.7 

6 

13 

MLS 

7 

7 

7 

Flaps 

7 3 

8 

14 

PME 

10 

9.5 

5 

Wind  Direction 

10 

9.5 

9 

Weather  Radar 

li  T 

11.5 

8 

»iind  Velocity 

n 3 

11.5 

9 

Barometric  Pressure 

13  7 

13 

9 

Outside  Air  Temperature 

’ 1 

14 

9 

it 

p 01 

r . .62 
p .02 

control 

RPV  Select 

1.3 

1 

2 

MLS 

4 

2 

3 

Electrical 

4.3 

3 

5 

AFCS 

4.7 

4 

13 

Hydraulic 

5.3 

5 

8 

Propulsion 

6 

6 

8 

FCS 

6.3 

7 

5 

Envi ronmenta 1 

7 

8 

8 

HAV  Mode 

6 

9 

13 

MFO 

10 

10 

11 

PME 

10.3 

11 

4 

MR 

11.3 

12 

5 

Transponder 

12.3 

13 

11 

Intercom 

14.3 

14 

1 

Headset 

V 

14.7 

15 

13 

\ 

M • .79 

p . .001 

r • .35 
n.s. 
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TABLE  17 


Sample  Work-Process  Chart 


CQMTftOlS  i 

1 

2 3 

• 

5 0 7 0 9 10  11 

12 

13 

14  15  16 

17 

18 

19 

FRCQUCNCV 

OF  use 

XtripMrf  Itid 

T 

2 

"5 

5 

5 

Vert<c«)  Speed  Ind. 

4.11 

1 

3 

4 

6 

7 

10 

AUiaetrr 

2 

2 

5.12 

3 

7.19 

7 

10 

19 

Aeder 

6.13 

4 

6.18 

8 

8 

20 

30 

LAOI  Dt&pUr 

3 

3 

4.10 

Id 

6 

5,17 

24 

4 

2 

11 

usi  CentreU 

0 

MSI  Ottpliy 

S 

S 

5,11 

3,7 

5 

4,16 

4 

13 

17 

11 

20 

£6T  tnd. 

7 

7 

7.14 

12,24  4 

10 

10 

21 

36 

RPM  tnd. 

8 

6 

3.10 

11.23  3 

9 

10 

17 

35 

Spoiler  Ind. 

9 

9 

2 

11 

4 

Speller  Cent 

1 

1 

flip  tnd. 

II 

14 

2 

Flep  Cent. 

11 

1 

MFO  Oispliy 

2 2.6  2,6  2.4,6  2.4.6 

9.21 

6.8 

10 

10  10 

33 

WD  Controls 

1 

1 1.5.9  1.5.9  1.3.5  1,3.5 

0.20 

32 

5,7 

1 ’« 

1 

Mode  Ind. 

r 

“3 

7 

h— 7 

AFS  Controlf 

2 

7 

5 

3 

W.J  Conlro\i 

0 

MIS  Ind. 

! 4 

0 

AOl  Oispliy 

4 

3.9 

IS 

4.0 

12 

3 

9 

iQJI  controls 

u 

Hcetner  Rjder  Oitp. 

0 

Neither  lUdir  Cent. 

0 

Audio  Nem  tnd. 

0 

Audio  Nim  Reset 

0 

Lendin^-Liqtit  Coflt. 

0 

Landlnp-lipM  Qlsp 

0 

Position  Plotter 

3.7 

13 

4 

11 

Fuel  Ind. 

0 

elect  c/d 

4.0 

3 

12 

Dl$t«nce-te-Go  Ind. 

5 

RPV-Select  Cent. 

1 

1.7 

1.8 

7 

13 

15 

Throttle  Cont 

2,9 

10.22  2 

0 

16 

34 

Cnd. -Gutdence  Cont. 

1? 

15 

2 

landInp-Gear  Cont. 

10 

10 

1 

12 

4 

drake  Cont 

0 

Nav/landinp  Mode  In« 

3.15  6 

4 

Joystick 

7.8 

27 

2 2.U 

7 

14 

26 

Oestruct  cent 

0 

Intereo* 

1 

13  1 

1 1.13 

25 

' 1 

10 

Transponder  Cont 

•I 

1 

Video  Controls 

1 

1 

1 

2 

Video  Otsplay 

2 

1 

iieadset 

0 

Radio  Cont. 

0 

Cowamicatlon 

COP 

COP 

COP 

COP 

COP  COP  COP  COP  COP  MC 

COP 

COP 

A/C  COP  COP 

COP 

COP 

RCF 

RCF 

A/C 

K COP 

Airspeed  Trial 

0 

Vertical-Speed  Trin 

0 

Radar-Alt.  Cont. 

0 

fAOl  Controls 

0 

Tl«  (sec.  1 

1 

M 

70 

10 

5 63  a«  76  51  117  29 

22 

42 

14  45  42 

21 

34 

45 

"T 

1 

00 

37 

1? 

30  41  35  23  25  8 

10 

31 

1 10  9 

13 

20 

5 
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obtained  for  Pilot  controls  and  displays  (p  <.01 ),  and  for  the  Copilot 
displays  (p  <.02).  The  correlation  between  the  Copilot  control  critical- 
ities and  frequency  of  use  was  not  significant  (p  > .10).  The  main  in- 
consistencies found  were: 

For  the  Pilot: 

1.  Although  the  MFD  display  and  controls  were  frequently  used, 
they  were  not  considered  critical  to  mission  success. 

2.  The  distance-to-go  indicator  was  given  a mid-range  rank  in 
terms  of  criticality;  however,  the  Pilot  never  referred  to  it 
during  the  tests. 

3.  The  intercom  was  the  most  frequently  used  control,  but  it  was 
ranked  twenty-second  in  terms  of  criticality. 

For  the  Copilot: 

1.  The  intercom  was  the  most  frequently  used  control,  but  it  had 
the  second  lowest  criticality  ranking. 

2.  The  PME  control  was  the  fourth  most  widely  used  control  but 
was  evaluated  eleventh  in  criticality. 


HUMAN  PERFORMANCE  ERRORS 


This  section  describes  the  major  errors  that  were  observed  during  test- 
ing. Error  consequences,  causes,  and  recommended  solutions/actions  are 
provided  for  each  error. 


Vehicle  Selection  Confusion 

Error  Description.  Both  the  Pilot's  and  Copilot's  consoles  have  two 
RPV-select  subpanels  (Figures  6 and  7).  One  set  of  vehicle  selection  push- 
buttons is  on  the  MFD  display-select  subpanel.  The  MFD  select  buttons  allow 
the  operator  to  select  a particular  vehicle's  data  for  display.  Additional 
pushbuttons  permit  selection  of  specific  subsystem  data.  The  second  set  of 
vehicle-select  pushbuttons  control  the  displayed  data  on  the  dedicated  in- 
struments in  each  console.  For  the  Pilot's  console,  this  second  control 
subpanel  (RPV  command  designate)  selects  the  vehicle  which  is  to  be  manually 
controlled  from  the  Pilot  station.  The  Pilot's  RPV  command  designate  sub- 
panel also  selects  the  vehicle  being  controlled  by  the  shared  subpanel. 
Fifteen  percent  of  the  time,  an  operator  pushed  a vehicle  select  button  on 
the  wrong  subpanel.  This  happened  more  frequently  at  the  Copilot  position 
when  an  operator  attempted  to  select  MFD  data  for  a specific  vehicle  by 
pushing  the  dedicated  display-select  control. 
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RPV  SELECT 
CONTROLS 


MFD  SELECT 
CONTROLS 


Figure  6.  Pilot  dual  RPV  select  designate  subpanels 
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MFD  SELECT 
CONTROLS 


RPV  SELECT 
CONTROLS 


Figure  7.  Copilot  dual  RPV  select  designate  subpanels 
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Error  Consequences.  For  the  Pilot,  the  consequence  of  actuating  the 
wrong  RPV  select  panel  sends  any  commands  (control  actuations)  made  after 
actuating  the  select  buttons  to  the  wrong  RPV.  This  act  could  have  cat- 
astrophic consequences  (e.g.,  collisions,  losing  control  of  a RPV,  etc.). 

If  the  Copilot  detects  the  Pilot's  error,  he  can  reselect  the  correct  RPV. 
However,  if  the  Copilot  fails  to  detect  this  error,  he  will  provide  the 
Pilot  with  incorrect  information  (e.g.,  the  Copilot  might  indicate  that  the 
generator  on  RPV  #2  was  malfunctioning  when  in  fact  it  was  RPV  #4  generator 
that  was  faltering). 

Recommended  Solutions.  The  alternative  solutions/actions  to  eliminate 
this  error  are: 

1.  Locate  the  MFD  control  panel  closer  to  the  MFD 

2.  Color  code  and  label  the  two  RPV  select  panels 

3.  Add  a display  next  to  the  RPV  select  panels  that  indicates 
which  RPV  is  being  controlled. 

The  solutions/actions  indicated  above,  along  with  other  solutions,  should 
be  evaluated  to  determine  the  most  cost  effective  procedure  for  eliminating 
this  error. 


Intercom  Panel  Confusion 

Error  Description  and  Consequences.  Each  operator's  intercom  panel 
includes  an  illuminated  TALK  and  a LISTEN  pushbutton  for  each  station  that 
can  be  addressed.  On  several  occasions  during  the  HFE  test,  an  operator 
mistakenly  pushed  a LISTEN  rather  than  a TALK  putton.  This  error  delayed 
communication  until  the  operator  pushed  the  correct  button. 

Error  Causes.  The  primary  reason  for  this  error  was  that  the  design 
included  both  TALK  and  LISTEN  buttons  for  communicating  with  a single 
station. 

Recommended  Solutions.  To  eliminate  this  problem,  the  feasibility  of 
redesigning  the  communication  panel  should  be  analyzed.  If  feasible,  the 
comnunication  panel  should  consist  of  a single  bank  of  illuminated  push- 
buttons, each  button  associated  with  a specific  station  (tower,  ATC,  Mission 
Control,  etc.).  Since  the  operators  must  use  their  hands  to  control  and 
monitor  multiple  RPV's,  a footswitch  should  be  provided  for  the  talk 
function.  If  a single  bank  of  pushbuttons  and  a footswitch  are  incorporated, 
the  advantages  will  be: 

1.  There  will  be  fewer  buttons  to  push. 

2.  The  error  of  pushing  the  LISTEN  rather  than  the  TALK  button 
will  be  eliminated. 

3.  A footswitch  will  reduce  requirements  for  using  hands. 
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Since  the  conmuni cation  function  is  one  of  the  most  frequently  used  sub- 
systems (see  Tables  15  and  16),  the  design  and  function  of  this  panel  should 
be  thoroughly  analyzed. 


Pilot  Overcontrol  of  Joystick 

Error  Description  and  Consequence.  Each  of  the  participants  tested 
had  a tendency  to  grasp  the  joystick  throughout  the  test  segments.  Figure 
8 shows  the  Pilot  actuating  the  joystick.  In  an  operational  setting,  this 
overcontrol  would  have  the  effect  of  continuously  adjusting  the  automatic 
pilot.  It  is  assumed  that  during  normal  conditions,  the  on-board  computer 
will  control  the  flight  parameters  of  the  RPV.  The  Pilot  does  not  need  to 
readjust  these  parameters  unless  some  condition  arises  that  necessitates 
a change. 

Error  Causes.  The  primary  reason  this  problem  occurred  was  that 
all  of  the  operators  were  used  to  taking  continuous,  active  control  of  air- 
craft, either  manned  or  unmanned. 

Recommended  Solutions.  It  is  expected  that  Pilots  will  be  selected  to 
become  RPV  controllers  and  that  they  will  have  a learned  predisposition  to 
hold  the  joystick  continuously.  If  so,  this  learned  behavior  must  be  trained 
out.  The  selected  operators  should  receive  training  on  when,  how  often,  and 
how  to  use  the  joystick. 

A slight  redesign  would  also  minimize  continued  handling  of  the  joystick. 
If  the  stick  were  mounted  on  a pedestal  which  extends  above  the  horizontal 
surface,  the  operator  would  be  less  likely  to  rest  his  hand  on  the  joystick. 


INCOMPATIBILITIES  AMONG  HUMAN  PERFORMANCE  AND  EQUIPMENT 


Task  Group  Interference 

The  only  task  group  interference  problem  observed  was  in  the  operator's 
use  of  a shared  panel.  A control  panel  located  between  the  Pilot  and  Copilot 
was  shared.  During  test  segment  performance,  both  operators  must  use  some 
of  the  controls  located  on  this  panel.  Figures  9 and  10  illustrate  Pilot 
and  Copilot  panel  use.  The  controls  located  on  the  panel  include: 

1 . Arm  and  Destruct  Mode 

2.  Flaps,  Spoiler,  and  Landing  Gear 

3.  Automatic  Flight  Control  System 

4.  Navigation  Mode 

5.  Landing  Mode 
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6.  Microwave  Landing  System 

7.  Command  Guidance 

The  Pilot’s  RPV-designate  control  panel  was  used  to  select  a particular 
RPV  for  manual  control.  Problems  arose  when  the  Copilot  attempted  to  change 
the  flight  parameters  of  a specific  RPV  before  the  Pilot  had  selected  that 
RPV  on  his  designate  console.  Because  of  this  error,  the  Copilot  altered 
the  flight  parameters  of  the  previously  selected  RPV  rather  than  change  the 
parameters  of  the  intended  RPV. 

The  immediate  cause  of  the  problem  was  that  the  operators  failed  to 
coordinate  their  activities.  The  Copilot  should  have  told  the  Pilot  that 
he  was  going  to  change  the  flight  parameters  of  RPV  #2  and  waited  for  the 
Pilot  to  confirm  that  it  was  safe  to  proceed  (RPV  #2  selected  on  RPV 
designate  panel).  Other  factors  contributing  to  this  error  were  equipment 
design  and  the  operators'  limited  experience  with  multiple  RPV  systems. 

The  shared-panel  problem  could  be  eliminated  or  substantially  reduced 
by  any  of  the  following  methods: 

1.  Allocate  control  to  a single  operator. 

2.  Display  designated  RPV  adjacent  to  or  on  shared  panel. 

3.  Provide  command-designate  panel  keyboard  for  shared  panel. 

4.  Train  operators  to  coordinate  their  activities  before  using 
the  shared  panel . 

If  the  control  panel  were  allocated  to  a single  operator,  the  shared 
panel  interference  problem  would  be  eliminated.  If  operator  loadings  prevent 
assigning  the  panel  to  a single  operator,  an  RPV-designate  display  should  be 
located  adjacent  to  or  on  the  panel  to  insure  that  an  operator  changes  only 
the  RPV  desired.  This  display  would  not  eliminate  the  problem  but  should 
markedly  reduce  its  occurrence. 

The  only  other  personnel  who  would  use  the  equipment  are  the  maintenance 
personnel.  If  any  major  component  in  the  RPV  control  station  malfunctioned, 
control  would  be  transferred  to  a backup  control  station.  The  component 
would  be  repaired/replaced  while  control  was  transferred. 


Equipment  Incompatibilities 

There  is  an  incompatibility  between  the  Runway  and  Glideslope  Monitor 
and  the  On-board  Video  Display.  The  on-board  video  camera  is  mounted  in  the 
nose  of  the  vehicle  with  a view  toward  the  runway  during  landing.  At  the 
same  time,  the  Runway  and  Glideslope  camera  is  mounted  at  the  departure  end 
of  the  runway  aimed  up  the  glideslope  toward  the  approaching  vehicle.  The 
cameras  look  directly  at  each  other,  so  they  have  completely  reversed  control/ 
display  relationships.  In  addition,  the  two  cameras  have  different  frames 
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of  reference.  The  Glideslope  camera  has  an  earth-coordinates  reference 
(frame  fixed  relative  to  earth)  and  the  on-board  camera  has  a vehicle- 
coordinates  reference  (frame  relative  to  RPV). 

The  incompatibilities  between  the  orientation  of  the  two  TV  cameras 
and  their  respective  frames  of  reference  will  cause  control  reversals 
and  hesitations  in  controlling  manually  the  vehicle's  roll  during  landing 
operations.  Such  control  reversals  can  have  catastrophic  consequences,  par- 
ticularly when  the  vehicle  is  at  low  altitude  on  the  landing  approach.  It 
is  strongly  recommended  that  one  of  the  two  displays  be  eliminated,  particu- 
larly since  the  displayed  information  is  highly  redundant. 

The  on-board  video  display  deserves  further  analysis,  particularly  if 
this  display  is  retained  and  the  runway  and  glideslope  monitor  is  eliminated. 
Extensive  discussions  and  research  have  addressed  the  question  of  which 
frame  of  reference  is  most  appropriate  for  aircraft  attitude  indicators.  An 
extensive  review  of  this  research  is  provided  by  Johnson  and  Roscoe  (16).  To 
sunmarize  their  major  findings,  an  earth-referenced  (moving  airplane)  display 
generally  produces  fewer  control  reversals,  particularly  if  the  operator  is 
located  in  an  earth-referenced  system  such  as  a fixed-base  simulator.  For 
the  present  vehicle  control  system,  problems  of  control  reversals  in  vehicle 
roll  can  be  reduced  if  the  on-board  TV  display  is  shown  as  a moving  airplane 
display,  as  it  would  be  with  an  earth  stabilized  camera.  The  motion  of  the 
displayed  vehicle  is  thus  controlled  by  feedback  signals  from  the  camera 
stabilization  system. 


Human  Performance  Problems 

A variety  of  console  configuration  problems  were  identified  by  perform- 
ing simulated  missions  with  the  mock-up.  These  problems  were  not  easily 
identifiable  by  merely  discussing  or  observing  console  drawings.  The  mock- 
up  and  simulated  missions  provided  a method  for  identifying  the  following 
console  configuration  problems. 

Missing  Controls  and  Displays.  During  training  and  simulated  missions, 
the  test  participants  reported  that  the  controls  and  displays  listed  next 
were  not  included  in  the  console  mock-up: 

1.  Oil  quantity  indicator 

2.  Flap  position  indicator 

3.  Fuel  pressure  indicator 

4.  Destruct  arm  indicator 

5.  Prime  mission  equipment  on/off  control/display 
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6.  Fuel  off  arm  control/display 

7.  Fuel  off  control/display 

8.  Takeoff-abort  control/display 

The  participants  stated  that  these  controls  and  displays  are  necessary 
for  successful  vehicle  operations.  The  validity  of  this  statement  should 
be  assessed,  and  it  should  be  determined  whether  the  operators  need  other 
controls  or  displays  to  accomplish  their  specified  human  performance  require- 
ments. 

Unnecessary  and  Redundant  Controls  and  Displays.  In  several  instances, 
separate  ON  and  OFF  pushbuttons  were  used  where  a single  ON/OFF  button 
(illuminated  in  the  ON  state)  would  have  been  appropriate.  This  was  the 
case  for  the  Vehicle  TV,  Microwave  Landing  System  (MLS),  and  Electronic 
Attitude  and  Direction  Indicator  (EADI)  control  subpanels.  In  the  case  of 
the  EADI  subpanel,  on  ON/OFF  control  is  not  absolutely  required,  since  there 
is  a series  of  mode  switches  (MLS,  SPD,  etc.),  one  of  which  must  be  on  at 
any  time.  Therefore,  pushing  any  of  these  mode  switches,  thereby  illuminat- 
ing the  mode  button,  could  also  be  used  for  the  ON  function. 

There  is  a row  of  status  or  mode  indicators  to  the  left  and  right  of 
the  EADI  display  on  the  Pilot's  console.  These  indicators  provide  readily 
accessible  flight  mode  and  subsystem  failure  information.  Many  of  these 
indicators  are  identical  with  display/control  indicators  on  the  Automatic 
Flight  Systems  (AFS)  and  Mode  Control  subpanel,  which  is  located  on  the 
shared  panel  between  the  Pilot  and  Copilot.  Since  the  indicators  on  the 
AFS  and  Mode  Control  subpanel  show  the  same  information  as  the  indicators 
next  to  the  EADI  display,  the  latter  indicators  should  be  eliminated.  The 
subsystem  failure  indicators  should  be  moved  to  the  subsystem  control  panels. 
Emergency  or  non-normal  conditions  should  be  indicated  with  flashing  push- 
buttons to  draw  attention  to  the  appropriate  control  subpanel  (36). 

Inappropriate  Control  and  Display  Placement.  From  an  analysis  of  the 
work  process  charts,  behavioral  checklists,  and  information  derived  from 
questionnaire  and  interview  data,  the  following  changes  are  suggested  for 
improving  the  locations  of  controls  and  displays  on  the  Pilot's  console. 

1.  Audio  warning  — move  out  of  prominent  panel  location.  There  is 
no  associated  visual  display,  and  a longer  reach  distance  to  the  reset  button 
will  not  be  disadvantageous. 

2.  Mission  time  clock  --  move  out  of  the  present  location  in  highly 
valuable  central  panel  space.  Since  the  Copilot  turns  the  clock  on  during 
takeoff,  it  could  be  placed  left  of  the  weather  radar  on  the  upper  panel. 

3.  ADI  — could  be  moved  from  the  central  panel,  since  it  serves  as 
a backup  for  the  EADI.  The  lower  center  location  would  then  be  available  for 
RPV  command  designate  controls. 
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4.  Multiple  vehicle  navigation  and  track  control  panel  --  this  panel 
is  located  on  the  horizontal  surface  directly  in  front  of  the  pilot.  This 
panel  includes  a number  of  pushbuttons  for  selecting  RPV's  and  for  control- 
ling the  electronic  plotter  map.  With  the  exception  of  the  RPV-select 
buttons,  the  controls  are  rarely  used.  In  addition,  placing  it  on  the  hor- 
izontal surface  intrudes  on  the  Pilot's  available  writing  space. 

The  console  designer  stated  that  a thorough  analysis  of  the  Copilot's 
station  had  not  been  undertaken  prior  to  testing.  Much  of  the 
Copilot's  workload  is  (1)  monitoring  the  status  of  multiple  vehicles,  (2) 
monitoring  subsystem  performance,  and  (3)  assisting  the  Pilot  during  take- 
off and  landing  by  calling  out  checklists  and  communicating  with  mission 
personnel.  Most  of  these  activities  should  involve  using  the  multi-function 
display  (MFD).  It  is  recommended  that  a thorough  analysis  be  made  of  the 
Copilot's  station  and  that  relocating  the  MFD  to  a more  central  location  be 
considered. 

Difficult  Cross-Correlation  Between  Operator  Panels.  During  testing  it 
was  observed  that,  on  occasion,  the  Copilot  would  lean  over  and  monitor  the 
Pilot's  controls  and  displays  to  cross-correlate  the  data  on  his  instruments 
with  those  of  the  Pilot.  Figure  11  shows  the  Copilot  monitoring  the  Pilot 
station.  No  particular  system  consequence  occurred  due  to  this  problem. 

If  this  cross-correlation  of  data  is  required,  this  problem  could  be  elim- 
inated by  either  slanting  the  Pilot's  and  Copilot's  stations  so  that  each 
could  easily  view  the  other  station;  or,  the  Copilot  could  request  that  the 
MFD  display  specific  Pilot  instruments. 


OBSERVED  SAFETY  HAZARDS 


No  safety  hazards  were  observed  during  testing,  and  no  safety  hazards 
were  reported  in  the  operators'  questionnaires. 


IMPACT  OF  HUMAN  PERFORMANCE  ON  ATTAINING  SYSTEM  PERFORMANCE  GOALS 


System  Performance  Goals 

The  Compass  COPE  system  requires  simultaneous  remote  control  of  up  to 
five  flight  vehicles.  Compass  COPE  is  capable  of  performing  four  distinct 
missions:  peripheral  reconnaissance,  battlefield  surveillance,  ocean 
surveillance  and  communications  relay.  The  system  performance  goals  call 
for: 
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Figure  11.  Copilot  monitoring  the  pilot  controls  and  displays 
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1.  Automatic  takeoffs  and  landings 

2.  Remote  control  of  vehicles  flying  to  the  target,  in  the  target 
area,  and  during  the  return  flight 

3.  Operation  in  the  required  high  altitude  and  long  endurance  (up 
to  thirty  hours)  missions. 


Impact  of  Problems  and  Errors 

The  data  from  this  HFE  test  indicate  that  the  tasks  assigned  to  the 
Pilot  and  Copilot  are  feasible.  However,  sufficient  problems  and  errors 
were  detected  to  indicate  that  it  will  be  difficult  to  control  the  Compass 
COPE  vehicles  reliably  without  modifying  or  refining  the  control  station. 
Although  the  identified  console  deficiencies  could  have  prevented  successful 
completion  of  an  actual  RPV  mission,  the  test  sequence  was  continued  beyond 
the  deficiency  point  by  assuming  that  the  missing  component  would  be  avail- 
able. This  procedure  permitted  completion  of  all  scheduled  test  events. 

As  described  previously,  a number  of  controls  and  displays  are  missing, 
redundant,  or  inappropriately  located;  some  controls  and  displays  produce 
errors  (e.g.,  pushing  similar  but  wrong  buttons),  and  at  least  two  incompat- 
ible displays  (On-board  Video  Monitor  and  Glideslope  Monitor)  are  expected 
to  produce  control  reversals.  The  anticipated  impact  of  these  error-producing 
design  problems  on  the  reliability  of  the  Compass  COPE  system  varies  from 
potentially  catastrophic  to  negligible. 

The  errors  which  appear  to  be  most  potentially  catastrophic  for  mission 
accomplishment  are  the  control  confusions  and  control  reversals.  Control 
confusion  arises  with  the  RPV  command-designate  and  MFD  vehicle-select  sub- 
panels in  which  an  operator  selects  a vehicle  on  one  subpanel,  intending  to 
use  the  displays  associated  with  the  other  subpanel.  As  described  previously, 
the  problem  is  particularly  acute  with  the  Pilot's  RPV  command-designate 
subpanel  since  this  selects  not  only  his  own  vehicle  control  displays  but 
also  the  shared  panel  which  can  be  operated  with  the  Copilot.  These  confusion 
errors  can  send  commands  to  an  unintended  vehicle  with  potentially  disasterous 
results,  particularly  since  there  is  no  guarantee  that  any  immediate  feedback 
would  prompt  the  operator  to  correct  his  error. 

Control  reversal  errors  are  expected  with  the  incompatible  On-board 
Video  Monitor  and  the  Glideslope  Monitor.  Although  immediate  visual  feedback 
is  available  from  either  display,  the  error  could  be  disasterous  since  the 
two  displays  are  used  simultaneously  during  the  landing  phase  when  the  RPV 
is  close  to  the  ground.  A reversal  in  roll  control,  when  the  vehicle  is 
within  a wing-span  distance  from  the  ground,  could  produce  a wing-tip  crash. 

Many  of  the  inappropriately  placed  displays  will  have  less  impact  on 
overall  system  reliability,  since  an  operator  usually  adapts  to  an  awkward 
arrangement  ^e.g.,  audio  warning,  mission-time  clock,  ADI).  However,  the 
awkward  arrangement  should  be  redesigned  to  eliminate  the  need  to  adapt  to 
it.  A similar  impact  on  reliability  can  be  expected  for  some  of  the 
redundant  controls  (e.g.,  separate  ON  and  OFF  buttons);  however,  the 
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unnecessary  duplication  of  TALK  and  LISTEN  buttons  may  cause  an  operator  to 
miss  a communication.  This  type  of  error  would  probably  not  be  catastrophic 
since  the  auditory  feedback  (or  lack  thereof)  is  so  immediate  that  it  stim- 
ulates error  correction.  Many  of  the  missing  controls  and  displays  are 
expected  to  be  mission-critical  (e.g.,  Takeoff-Abort  control/display,  fuel 
pressure  indicator,  fuel  off  and  fuel  off  arm  control /displays,  etc.).  Other 
missing  items  are  probably  not  mission-critical  but,  rather,  are  convenient 
or  expected  since  they  have  been  included  in  previous  aircraft  cockpit 
designs  (e.g.,  flap  position  indicator).  Some  displays  which  were  noted  as 
missing  may  not  require  addition  to  the  control  panels,  since  the  information 
could  be  programmed  for  display  on  the  MFD. 

If  Pilots  overcontrol  the  joystick  there  will  probably  be  little  effect 
on  reliability,  since  immediate  visual  feedback  from  the  EADI  will  signal  the 
Pilot  that  he  is  making  a control  input.  The  lack  of  cross-referencing  be- 
tween the  Pilot  and  Copilot  display  panels  may  have  moderate  impact  on  system 
reliability,  since  an  operator  may  be  too  involved  to  look  over  to  the  other 
panel  to  obtain  potentially  critical  information. 

One  major  area  of  human  performance  could  not  be  assessed  during  testing, 
"amely,  an  operator's  ability  to  interpret  and  use  information  displayed  on 
the  multifunction  display  (MFD).  The  impact  of  human  performance  with  this 
display  should  be  studied  before  system  design  progresses  significantly.  It 
is  recommended  that  a human  performance  test  be  conducted  as  soon  as  dynamic 
simulation  of  the  Ground  Control  System  (GCS)  computer  functions  is  available. 
This  simulation  can  be  conducted  on  the  present  mock-up  by  adding  computer- 
generated CRT  displays  to  simulate  MFD  information. 


Impact  Upon  System  Effectiveness 

Error  frequencies  and  performance  times  from  the  static  mock-up  of  the 
experimental  prototype  of  the  Ground  Control  Station  will  probably  have  little 
relationship  to  the  frequencies  and  times  in  the  final  system.  Static 
displays  provide  no  feedback  for  error  correction  nor  for  effective  evalu- 
ation of  display  monitoring  time.  Therefore,  system  effectiveness  was  not 
estimated  on  the  basis  of  these  test  data. 


Solutions  for  Improving  Human  Performance 

The  majority  of  recommended  solutions  to  the  problems  and  errors  noted 
in  this  test  involve  modifying  the  operator  station's  design.  Design  changes 
are  particularly  advantageous  over  training  or  personnel  selection  solutions 
at  this  time  because  (1)  in  most  cases,  a redesign  should  eliminate  the 
problem  rather  than  reduce  its  occurrence,  (2)  a design  change  is  a non- 
recurring cost,  and  (3)  this  early  stage  of  system  development  encourages 
redesign,  rather  than  changing  a training  or  personnel  selection  program 
which  has  yet  to  be  defined. 

As  described  previously  in  the  discussion  of  each  problem  or  error,  a 
redesign  is  expected  to  eliminate  problems,  thereby  increasing  the  probability 
of  successful  mission  completion.  Redesign  will  be  particularly  important 
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for  the  confusion-producing  RPV  command-designate  subpanels  and  the  shared 
panel  command  function,  the  incompatible  Glideslope  Monitor  and  On-board 
Video  Monitor,  and  some  of  the  missing  controls  and  displays.  Since  these 
problems  are  expected  to  have  potentially  catastrophic  effects,  eliminating 
the  problems  will  obviously  improve  system  reliability  and,  thereby,  effec- 
tiveness. These  design  recommendations  can  be  implemented  in  an  economical 
manner  since  the  Compass  COPE  ground  station  is  still  in  an  early  design 
stage.  The  resulting  improvement  in  human  performance  is  expected  to  have 
a major  impact  on  overall  system  success. 


CONCLUSIONS 


Test  Findings 

The  major  HFE  test  findings  include  the  following  items: 

A.  Problems: 

1.  Some  necessary  controls  and  displays  were  not  included 

2.  Redundant  or  unnecessary  controls  and  displays  were  included 

3.  Several  controls  and  displays  were  inappropriately  placed 

4.  Operators  experienced  difficulty  cross-referencing  with  other 
operator's  panel. 

B.  Performance  Errors: 

1.  Operators  used  wrong  controls  in  selecting  vehicles 

2.  Pilot  over-controlled  joystick. 

C.  Equipment  and  Task  Group  Incompatibilities: 

1.  On-board  Video  and  Glideslope  Monitors  are  incompatible 

2.  Operators  interfered  with  each  other  in  using  the  shared 
subpanel . 


Implications  for  System  Performance 

The  major  implications  of  this  HFE  test  for  system  performance  are  that, 
without  design  modi ficationb , the  multiple  RPV's  probably  cannot  be  success- 
fully controlled.  The  control  confusion,  shared  panel  task  group  interfer- 
ence, incompatible  displays,  and  som-?  of  the  missing  controls  and  displays 
are  expected  to  produce  catastrophic  failures.  The  other  problems  and  errors 
are  expected  to  reduce  system  reliability  and  availability. 
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Recommended  Changes 


The  following  changes  are  recommended  to  eliminate  sources  of  error  and 
to  improve  system  reliability  and  effectiveness: 

1.  Redesign  RPV  command-designate  subpanels  (MFD  and  RPV  select) 
to  eliminate  confusion  between  them. 

2.  Allocate  the  present  shared  panel  to  a single  operator,  or  design 
the  shared  panel  to  display  the  selected  vehicle. 

3.  Eliminate  one  of  the  incompatible  displays  (probably  the  Glide- 
slope  Monitor) . 

4.  Use  earth-referenced  (moving  airplane)  coordinates  in  EADI  and 
On-board  Video. 

5.  Include  critical  missing  controls  and  displays. 

6.  Eliminate  redundant  and  unnecessary  controls  and  displays. 

7.  Place  controls  and  displays  according  to  criticality,  frequency, 
and  sequence-of-use  data. 

8.  Adjust  the  angle  between  operator  panels  to  facilitate  cross- 
referencing. 
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REPORT  OF  HUMAN  FACTORS  TEST  OF  A SYSTEM  IN  ADVANCED  DEVELOPMENT 
(COMMUNICATION  CONTROL  UNIT  OF  TACFIRE) 


This  is  a sample  of  an  HFE  test.  The  purpose  of  this  test  report  is  to 
provide  guidelines  for  conducting,  analyzing,  and  reporting  on  HFE  tests  for 
systems  in  advanced  development.  The  Communication  Control  Unit  (CCU) 
operator  is  the  subject  of  the  test  evaluation.  This  report  is  an  evaluation 
of  the  task  requirements,  training,  selection  criteria,  and  interface 
equipment  pertaining  to  that  position. 

This  report  illustrates  the  contents  and  format  for  a test  document 
prepared  according  to  the  specifications  of  DI-H-1334A, 


NOTICE 


TACFIRE  is  currently  in  the  Low  Rate  Initial 
Production  phase  of  engineering  development. 
However,  for  illustrative  purposes,  it  was 
assumed  that  this  test  was  conducted  at  the 
end  of  advanced  development  and  prior  to  any 
design  freeze. 
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INTRODUCTION 


Task  Group  Identification 

This  report  describes  the  procedures  and  results  of  a human  factors 
engineering  (HFE)  test  of  the  Conmunication  Control  Unit  (CCU)  operators' 
task  group  of  the  U.S.  Army  TACFIRE  artillery  fire  direction  system.  This 
operator  is  responsible  for  establishing  and  maintaining  communication 
links  among  the  personnel  and  digital  data  equipment  of  TACFIRE  units  in 
field  artillery  battalions  and  divisions.  Field  Artillery  Cannon  Operations/ 
Fire  Direction  Assistants  (MOS  13E)  are  the  personnel  who  will  be  selected 
to  operate  the  CCU. 


CCU  Operator  Interface  Equipment 

The  CCU  operator  uses  the  Cotmunication  Interface  Equipment  (CIE)  to 
(1)  establish  nets  among  multiple  subscribers  and  Digital  Data  Terminals 
(DDTs),  and  (2)  establish  links  between  selected  subscribers.  This  equipment 
provides  monitoring,  switching,  control  and  interface  capabilities  between 
various  items  of  TACFIRE  communication  equipment.  The  CIE  inventory  includes 
the  Communication  Control  Unit  (CCU),  and  the  Conmunication  Terminal  Box 
(CTB). 

ecu.  The  CCU  is  the  primary  equipment  for  switching  and  controlling 
TACFIRE  conmuni cat ions.  The  CCU  provides  the  capability  for  establishing 
wire  and  radio  conmunication  links  for  voice  and  digital  data.  The  device 
utilizes  microprocessor  capabilities  to  replace  some  of  the  hardware 
functions  of  the  earlier  TACFIRE  device  (monitoring,  patching,  and  control 
unit).  The  CCU,  shown  in  Figure  12,  is  housed  in  a 23"  high,  31"  deep,  and 
18"  wide  type  C transit  case.  The  total  weight  of  the  unit  is  150  pounds. 

The  major  components  of  the  CCU  are  a display  panel,  alphanumeric  keyboard, 
phone  jacks,  and  connectors. 

CTB.  Before  the  CCU  operator  can  use  the  CCU's  capabilities,  wire 
1 ines  from  radio  and  wire  subscribers  must  be  connected  to  the  Communication 
Terminal  Box  (CTB).  The  CTB  is  a completely  self-contained  unit  housed  in 
a transportable  case  46.75  inches  high,  18.13  inches  wide  and  9 inches  deep. 
The  unit  weighs  135  pounds.  The  assembly  is  normally  mounted  to  the 
shelter's  exterior  surface  near -the  door  entry.  As  shown  in  Figure  13,  the 
CTB  has  a total  of  144  spring-actuated  binding  posts  for  connecting  field 
lines.  The  binding  posts  and  connector  pins  are  wired  in  parallel.  The 
CTB  is  a passive  unit;  no  power  is  required  for  its  operation. 


Test  Site 

The  test  was  conducted  between  26  and  31  January  1976,  at  the  ARTADS 
Field  Office,  Ft.  Sill,  Oklahoma.  Due  to  equipment  unavailability  during 
normal  working  hours,  test  sessions  were  conducted  daily  between  0130  and 
0700.  During  testing,  the  participants  were  excused  from  their  normal  duties 
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Figure  12.  Communication  Control  Unit 


MICROCOPY  RESOLUTION  TEST  CHART 

NATIONAL  BURLAU  OF  STANDARDS  196J  A 


this  test  was  their  only  assigned  duty.  The  tests  were  supervised  and 
conducted  by  Dr.  William  H.  Crooks  and  Mr.  Barry  L.  Berson,  of  Perceptronics, 
Inc. 


TEST  PREPARATION 


ecu  Operator's  Task  Group 

An  analysis  of  the  human  performance  requirements  for  the  CCU  operator's 
task  group  indicated  that  the  operator  was  required  to  perform  the  following 
five  primary  tasks: 

1.  Connect  subscribers  to  the  CTB. 

2.  Perform  equipment  start-up  procedures. 

3.  Connect  subscribers  to  nets. 

4.  Perform  CCU  operational  duties. 

5.  Perform  CCU  operational  level  maintenance. 

The  human  performance  analysis  gave  a complete  sequential  list  of  all 
the  tasks,  subtasks,  and  steps  in  the  CCU  operator's  task  group  (Table  18). 


Human  Performance  Standards 

No  human  performance  standards  have  been  developed  for  the  CCU  operator. 


Test  Environment 

The  HFE  test  was  conducted  in  a standard  TACFIRE  Battalion  Electrical 
Equipment  Shelter  (Model  S-280),  which  had  been  cross-sectioned  as  illustrated 
in  Figure  14.  This  van  was  located  in  a large  classroom  at  the  Ft.  Sill 
facility.  Therefore,  the  operator's  work  station  was  identical  to  the 
operational  environment,  except  that  light  from  the  classroom  increased  the 
van's  normal  illumination,  and  the  operator  had  more  seating  room  than  in  a 
normal  shelter.  To  compensate  for  the  latter  condition,  the  operator  was' 
not  allowed  to  move  his  chair  back  farther  than  the  twenty-five  inches 
normally  available. 

Components  from  the  TECOM  HFE  Instrument  package  were  used  to  measure 
environmental  conditions.  Each  measure  was  recorded  on  the  first  and  last 
nights  of  testing.  Environmental  measures  were  not  recorded  more  frequently 
because  the  test  was  conducted  in  an  indoor  facility  and  conditions  at  the 
test  site  did  not  change  significantly  during  the  test.  The  measures  were 
replicated  during  the  last  night  of  testing  to  verify  that  the  environment 
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TASK  1 


TASK  2: 


TASK  3; 


TABLE  18 

ecu  Operator's  Task  Group 


: coma  SUeSCAIBEAS  TO  the  commicatiqns  tewipial 
BOX  (CTB) 

SubUtks 

1.1  Connect  til  rodio  lubscrlbtrt  lUtod  on  tht 
tubtcriber  directory  to  the  CTB 

1.2  Connect  all  local  lubscrlbert  listed  on  the 
subscriber  directory  to  the  CTB 

1.3  Connect  all  wire  subscribers  listed  on  the 
subscriber  directory  to  the  CTB 

; 9tW<m  EqUIP»«HT  STARTOP  P1WXE0UWS 

Subtasks 

2.1  Adjust  bright  control  to  Midrange 

2.2  Adjust  display  bright  control  to  nidrange 

2.3  Adjust  1a^>  bright  control  to  Midrange 

2.4  Adjust  speaker  i^oIums  control  to  Midrange 

2.5  Adjust  switchboard  volune  control  to  Midrange 

2.6  Set  alert  switches  to  headset  position 

2.7  Set  power  on/off  switch  to  on 

2.8  Press  keyboard  clear  key  two  tlaes  or  until 
window  Is  blank 

2.9  Set  radio  m/fm  switch  to  correct  position 
for  each  radio  connected  to  the  CTB 

CONNECT  SUBSCRIBERS  TO  NHS  INDICATED  ON  THE  NET 
CONPiejARATION  DATA  FORM 

Subtasks 

3.1  Set  up  net  1 

a)  Press  NET  key 

b)  Press  1 key 

e]  Add  radio  to  net>-on1y  If  radio  stibscriber 
Is  to  be  placed  on  net 

1)  Press  radio  (RAO)  key 

2)  Press  radio  nwber  key 

3)  Press  connect  (CONN)  key 

4)  Repeat  steps  1 through  3 for  the 
reaelnlng  radios  (up  to  3 per  net)  to 
be  connected  to  net  1 

d)  Add  wire  to  net— only  If  wire  subscriber  Is 
to  be  placed  on  net 

1)  Press  HIRE  key 

2)  Press  wire  nuiter  key 

3)  Press  connect  (CONN)  key 

4)  Repeat  steps  1 through  3 for  the 
reaelnlng  wire  subscribers  desired  on 
net  1 (up  to  6 per  net) 

e)  Add  dieltal  data  tererinal  (DOT)  to  net- 
only  If  a DOT  Is  to  be  connected  to  the  net 

1)  Press  DOT  key 

2)  Press  DOT  nta^er  key 
I)  Press  CONN  key  (only  one  DOT  per  net) 

f)  Add  iMal  (LCL)  subKrIber  to  net-'Only  If 
a local  subscriber  Is  to  be  connected  to  the  net 

1)  Press  LCl  key 

2)  Press  LCL  niM^er  key 

3)  Press  CONN  key 

4)  Repeat  steps  t through  3 for  the  reaelnlng 
iKal  subscribers  desired  on  the  net  (up  to 
twenty  per  net) 

3.2  Sat  up  ether  nets 

a)  Repeat  all  of  the  steps  in  3.1  for  each  of  the 
other  nets  Indicated  on  the  net  configuration 
fora. 


TASK  4:  PERFORM  CCU  OPERATIONAL  DUTIES 

Subtasks 

4.1  Monitor  volct/data  traffic  on  net  1 

a)  Press  Net  1 key 

b)  Press  Monitor  (NON)  key 

4.2  Talk  on  NET 

a)  Press  TALK  key 

b)  Press  the  PUSH-TO-TALK  switch  on  the 
headset 

c)  Coaeunicata  with  desired  subscriber 

d)  Press  OFF  key  to  teralnate  conversation 

4.3  Answer  net  call 

a)  Press  answer  (ANS)  key 

b)  Press  the  headset  PU$H-TO-TALK  switch 

c)  Answer  call 

d)  Press  OFF  key  to  teralnate  conversation 

4.4  Link  subscribers 

a)  Press  LINK  key 

b)  Press  the  appropriate  link  nunber 

c)  Input  subscriber  type 

d)  Input  subscriber  ntwber 

a)  Press  connect  (CONN)  key 

f)  Repeat  steps  a to  e for  each  subscriber 
to  be  linked 

4.5  Disconnect  subscribers  free  link 

a)  Press  LINK  key 

b)  Press  link  nuMber 

c)  Press  the  disconnect  (DISC)  button 
three  tlaes 

4.6  Add/delete  subscriber  frea  a Net 

a)  Press  net  nunber 

b)  Press  subscriber  type 

c)  Press  subscriber  nunber 

d)  Press  CONN/DISC  depending  upon  whether 
the  subscriber  was  to  be  added  or  reaoved 
froM  the  net 

TASK  5:  PERFORM  OPERATIONAL  LEVa  MAINTENANCE  ON  THE  CCU 

Subtasks 

5.1  Perfora  wire  fault  Isolation  test  on  selected 
subscribers 

a)  Press  keyboard  clear  (CLR)  key  two  tlaes 
or  until  error  window  Is  blank 

b)  Press  LAMP  button  and  Monitor  CCU  display 

c)  Press:  NET,  net  nuaber,  subscriber 
noiber.  CONN 

d)  Press  PUSK>TO-TALK  switch  to  cenaunicate 
with  desired  subscriber 

e)  Press  OFF  key  to  discontinue  coaaunicatlon 

f)  Repeat  steps  a through  a for  each 
subscriber  connected  to  the  CTB 

5.2  Perfora  CCU  card  checkout  on  selected  cards 

a)  Press  keyboard  clear  (aR)  key  two  tines 
or  until  error  window  It  blank 

b)  Press  TEST  key 

c)  Press  subscriber  type 

d)  Press  subscriber  nuiber 

e)  Press  Ring  key 

f)  Monitor  error  window.  If  a 79  appears  In 
the  error  window,  the  card  Is  no  good;  if 
an  BO  appears  In  the  window,  the  card  Is  OK 

g)  Repeat  steps  a through  f for  each  selected 
card 


was  stable.  Table  19  presents  the  means  and  standard  deviations  of  the 
environmental  measures.  The  small  standard  deviations  demonstrate  the 
stability  of  the  environment.  Figure  15  shows  the  measured  octave  band 
noise  levels. 


Test  Participants 

Five  operators,  four  Anny  TACFIRE  cadre  personnel  and  one  civilian 
from  the  U.S.  Ant\y  Electronics  Command  (ECOM),  served  as  the  participants 
for  this  HFE  test.  The  decision  to  use  five  participants  was  based  on  the 
stage  of  system  development,  the  expected  variability  of  test  participant 
performance,  and  budgetary  constraints.  Five  participants  were  deemed 
adequate  to  provide  the  amount  and  type  of  data  required  for  a meaningful 
analysis  of  test  results.  Test  participant  characteristics  are  shown  in 
Table  20. 

Anthropometric  dimensions  for  these  participants  are  given  in  Table 
21,  which  also  includes  normative  dimensions  of  the  5th,  50th,  and  95th 
percentile  of  military  populations.  As  shown  in  this  table,  all  test  par- 
ticipants were  in  the  low  to  middle  ranges  of  all  dimensions. 


Test  Participants'  Clothing  and  Personal  Equipment 

The  four  military  personnel  were  dressed  in  fatigues.  The  civilian 
operator  wore  a sport  shirt  and  slacks.  During  testing,  the  participants 
used  either  a loudspeaker  and  a hand-held  microphone,  or  a communication 
headset  (CH-182/U).  No  protective  masks  were  worn,  since  the  operators  are 
expected  to  work  in  an  environmentally  protected  shelter. 


Participant  Training 

Each  of  the  four  military  operators  had  received  eight  weeks'  training 
on  TACFIRE  operation  and  maintenance  procedures  approximately  three  months 
prior  to  testing.  Of  the  eight-week  period,  approximately  two  days  were 
devoted  to  CCU  operation  and  maintenance.  Each  participant  received  approx- 
imately thirty  minutes  of  "hands  on"  experience.  The  ECOM  civilian  had 
about  one-and-a-half  years  of  experience  with  the  CCU.  During  the  time 
between  the  participants'  initial  training  and  their  refresher  training, 
they  were  not  assigned  to  operate  the  CCU  and  received  no  practice  performing 
CCU  operational  tasks. 

About  twenty-four  hours  prior  to  testing,  the  participants  were  given 
a one-hour  refresher  lecture/discussion  on  CCU  operation  and  maintenance 
procedures.  During  this  time,  all  of  the  tasks  that  the  operators  were 
required  to  perform  during  the  test  were  discussed,  including  the  procedures 
for: 
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TABLE  19 


Environmental  Recordings 


I 


ENVIRONMENTAL  RECOROING 

MEAN 

STANDARD  DEVIATION 

1. 

Air  Flow 

a.  At  ecu  Keyboard 

70  Feet/Minute 

7.1 

b.  At  ecu  Display 

190 

14.1 

c.  Maximum  Flow  - About  2 ft. 

925 

70.7 

Over  ecu  Operator's  Seated 
Height 

2. 

Illumination 

a.  At  ecu  Display  Surface 

S.7  Ft.  Candles 

.28 

b.  At  ecu  Keyboard  (Lamps 

10.8 

1.1 

Full  Bright! 

c.  Chest  Level  (In  Front  of 

17.3 

1.1 

ecu  Display) 

3. 

Spot  Brightness 
a.  Single  LEO  Number 

.55  Ft.  Lamberts 

.04 

b.  Display  Background 

c.  Keyboard  (Individual  Key) 

.05 

6.4 

.01 

.57 

d.  Keyboard  Surround 

1.25 

.26 

e.  Single  Letter  on  a Key 

5.2 

.43 

4. 

Temperature 

a.  In  Test  Shelter 

22“C 

5.4 

N 

2 

2 

2 

2 

2 


10 


5. 

Relative  Humidity 
a.  In  Test  Shelter 

35X 

6. 

Steady  State  Noise 
a.  Weighting  A 

68 

b.  Weighting  B 

69 

c.  Weighting  C 

72.5 

d.  Flat 

79.5 

2.8  10 


db  1.4  2 

" 1.1  2 

.71  2 

" 1.4  2 
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CSJ  <M  CM  csi  C\J 


A 


a> 


>» 


Center  Frequency  (Hz) 


Figure  15.  Measured  octave  band  noise 
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PARTICIPANT 


CHARACTERISTIC 

1 

2 

3 

4 

5 

AGE  (years) 

27 

36 

28 

25 

21 

WEIGHT  (pounds) 

195 

130 

200 

130 

130 

SEX 

M 

H 

H 

F 

M 

SCHOOLING 

3- 

M.A. 

12- 

8. A. 

3- 

(highest  year) 

College 

HS 

College 

YEARS  IN  SERVICE 

5 

Civilian 

9 

1 

.75 

SERVICE  GRADE 

E-4 

N/A 

E-6 

E-5 

PFC 

MONTHS  IN  GRADE 

29 

N/A 

60 

8 

9 

VISUAL  ACUITY 

20/20 

20/20 

20/20 

20/20 

20/20 

DISABILITIES 

NONE 

NONE 

NONE 

NONE 

NONE 

AGCT  SCORE 

142 

N/A 

125 

123 

135 

ecu  TEST  SCORE 

80% 

93% 

79% 

74% 

97% 

TABLE  21 


Test  Participant  Anthropometric  Measurements 


BODY  DIMENSIONS 
(INCHES) 

1 

P 

2 

A R T I 

3 

C I P 

4 

STANDING  HEIGHT 

69 

70 

72 

64 

SITTING  HEIGHT 

35.5 

32 

35 

34.5 

EYE  HEIGHT 

31.5 

28.5 

30.5 

29 

ELBOW  RESTING  HEIGHT 

10.5 

7 

10 

9.5 

HORIZONTAL  ARM  REACH 

32.5 

31 

34 

30 

VERTICAL  ARM  REACH 

53 

50 

53 

47 

ELBOW-FINGERTIP  REACH 

18 

18 

18.5 

16 

BUTTOCK-HEEL  LENGTH 

43 

42 

45 

40 

* From  VanCott  and  Kinkade,  1972 


NORMATIVE  DIMENSIONS® 
(PERCENTILE  GROUPS) 


5 

Mean 

5% 

50% 

95% 

70 

69 

64 

67 

75.2 

34.5 

34.3 

32.5 

35.8 

38.2 

29.5 

29.8 

28.8 

30.9 

33.1 

8.5 

9.1 

7.4 

9.5 

11.6 

32 

31.9 

31.1 

33.7 

36.3 

51.5 

50.9 

51.6 

55 

59 

18.5 

17.8 

17.3 

18.7 

20.1 

45 

43 

39.8 

44.3 

48.4 

71 
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1.  Connecting  subscribers  to  the  CTB 

2.  Setting  up  nets 

3.  Using  the  subscriber  directories 

4.  Performing  CCU  operations 

5.  Performing  CCU  operational  maintenance 

Immediately  following  the  discussion,  each  participant  received  one  and  one- 
half  hours  of  practice  in  performing  CCU  operation  and  maintenance  tasks. 

To  assess  the  test  participants'  knowledge,  a CCU  operation  and  main- 
tenance procedures  test  was  administered  to  the  participants  before  testing. 
The  test  consisted  of  thirty  questions,  fifteen  taken  from  the  initial  CCU 
training  course  test,  and  fifteen  new  questions  derived  from  the  human 
performance  analysis.  Test  scores  ranged  from  74  to  97%  correct.  Test 
score  results,  together  with  the  ECOM  representative's  opinion  that  the  test 
participants  were  adequately  trained,  were  used  to  determine  that  the 
participants  were  ready  to  begin  testing. 


Test  Equipment 

The  HFE  test  used  an  operational  Communication  Control  Unit  and 
Communication  Terminal  Box.  To  simulate  TACFIRE  subscribers,  phones,  radios, 
and  another  CCU  were  connected  to  the  CTB  with  wire  lines.  After  the  test 
operator  connected  the  wire  lines  to  the  CTB,  he  could  establish  voice 
communications  with  the  subscribers. 


Deviations  of  Test  from  Operational  Conditions 

The  major  difference  between  the  test  and  operational  conditions  was 
using  cadre  level  personnel  as  participants.  These  participants  received 
more  training  (seven  hours  of  lectures  and  seven  hours  of  "hands-on"  practice) 
than  the  operation  personnel  will  recieve  (three  and  three).  It  is  assumed 
that  if  personnel  with  only  three  hours  of  lecture  and  three  hours  of  "hands- 
on"  training  were  tested,  they  would  have  committed  more  errors. 

The  following  deviations  between  test  and  operational  conditions  also 
occurred: 

1.  Brighter  lighting  in  the  test  shelter 

2.  More  leg  room  for  test  participants 

3.  Fewer  personnel  in  the  shelter  (during  operations  as  many  as 
four  persons  will  perform  their  tasks  in  the  shelter). 

The  first  two  deviations  were  caused  by  using  the  cross-sectioned  van. 
Both  of  these  deviations  were  controlled.  A black  sheet  was  placed  over  the 
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test  shelter  to  reduce  the  illumination  level,  and  the  operator  was  not 
allowed  to  move  his  chair  back  farther  than  the  maximum  distance  allowable 
under  operational  conditions.  The  third  deviation  was  not  controlled  but 
it  is  assumed  that  it  did  not  significantly  affect  the  test  data. 


DATA  COLLECTION  TECHNIQUES 


Test  Plan 

The  test  plan  called  for  testing  each  of  the  five  participants  on  each 
of  the  five  major  CCU  operator  functions.  Five  sessions  were  conducted. 
During  each  session,  one  operator  was  tested  at  each  of  the  five  major  tasks. 
Each  session  took  about  two  and  one-half  to  three  hours  to  complete.  Par- 
ticipants were  randomly  assigned  to  sessions. 


Data  Collection  Methods  and  Equipment 

Behavioral  checklists  were  used  to  facilitate  the  process  of  collecting 
and  recording  task  group  time  and  error  data  (see  Appendix  2).  Checklists 
were  developed  for  each  major  operator  task.  Each  cnecklist  contains: 

1.  A sequential  list  of  all  operator  tasks  and  subtasks 

2.  Descriptions  of  required  operator  responses  for  each  task 

3.  Task  success  criteria  (CCU  display  indicating  satisfactory 
performance) 

4.  Columns  for  recording  task  response  duration,  response  adequacy 
(correct/error),  and  error  descriptions 

Tasks  on  the  checklists  were  arranged  in  the  same  order  that  the  operator 
performed  them.  To  evaluate  an  operator's  response  on  any  task,  the  test 
director  would: 

1.  Observe  the  operator's  response 

2.  Observe  the  CCU  display  after  the  response 

3.  Compare  these  observations  to  the  response  required  and  the 

success  indicator  listed  on  the  checklist. 

If  the  operator  performed  the  task  correctly,  the  test  director  would 
place  a check  (/)  in  the  appropriate  column  of  the  checklist.  If  the  operator 
erred,  the  test  director  would  record  a brief  description  of  the  error  in 
the  appropriate  column. 
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A Heathkit  digital  stopwatch  was  used  to  measure  operator  response  times. 
The  stopwatch  was  started  when  the  test  director  told  the  operator  to  begin 
and  was  stopped  when  the  operator  indicated  that  the  task  was  completed.  Task 
response  times  were  recorded  on  the  behavioral  checklist. 

A Panasonic  black  and  white  video  tape  recorder  (VTR)  was  used  to  obtain 
a permanent  record  of  observed  task  group  problems  and  errors.  The  VTR  was 
also  used  to  obtain  documentary  footage  of  the  test. 


Test  Procedures 

This  subsection  describes  the  procedures  used  to  collect  operator  per- 
formance data  for  each  of  the  five  tasks. 

CTB-Subscriber  Connection.  For  this  task  segment,  the  operators  were 
given  a preassigned  subscriber  directory  (Table  22).  This  directory  showed 
the  type  of  circuits  to  be  connected  and  the  name  of  the  subscriber  asso- 
ciated with  each  circuit.  The  operators  were  told  to  connect  all  of  the 
subscribers  listed  in  the  directory.  They  were  also  told  to  work  as  fast 
as  they  could  without  making  errors.  In  all,  thirty-five  circuits  were 
connected.  After  the  operators  connected  the  circuits,  the  test  director 
used  a template  (Figure  16)  to  determine  if  all  of  the  circuits  were  connected 
correctly. 

Equipment  Start  Up.  In  this  segment,  the  operators  were  instructed  to 
adjust  all  display,  headset,  and  speaker  controls  to  midrange,  to  turn  on 
the  ecu,  and  to  clear  the  keyboard.  The  test  director  observed  and  recorded 
operator  performance  on  the  checklist. 

Net  Set  Up.  A printed  net  directory,  illustrated  in  Figure  17,  was 
given  to  each  operator.  The  entries  in  the  form  were  assigned  by  the  test 
director  prior  to  the  test  session.  This  directory  listed: 

1.  DDT  number,  if  any,  associated  with  each  net 

2.  The  number  and  name  of  each  radio,  local,  and  wire  subscriber 
to  be  placed  on  each  net. 

The  operators  were  required  to  set  up  the  nets  as  listed  on  the  directory. 

The  time  required  to  set  up  each  net  was  recorded  on  the  checklist.  After 
the  nets  were  set  up  the  test  director  instructed  the  operators  to  display 
each  net.  The  test  director  compared  the  subscribers  placed  on  each  net  with 
the  subscribers  on  the  checklist.  If  subscribers  had  not  been  placed  on  the 
correct  net,  the  test  director  told  the  operator  to  add  them  to  the  net 
Likewise,  if  subscribers  were  placed  incorrectly  on  nets,  the  subscribers 
were  adjusted  accordingly.  This  adjusting  procedure  continued  until  all 
subscribers  were  connected  correctly.  Subscriber  placement  er.ors  were 
recorded  on  the  checklist. 
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TABLE  22 


Subscriber  Directory 


CIRCUIT  TYPE 

SUBSCRIBER 

CIRCUIT  TYPE 

SUBSCRIBER 

Radio 

Local 

* 1 

FM 

Remote  Radio  1 

* 1 

Bn  Cdr 

* 2 

FM 

Remote  Radio  2 

* 2 

Mess 

3 

FM 

Remote  Radio  3 

3 

Supply 

4 

AM 

Bdg  FSO 

4 

Motor  Pool 

*'5 

AM 

Bn  CDR 

* 5 

Security 

6 

FM 

FSO  1 

6 

Station  1 

7 

FM 

Batt  A 

7 

Station  2 

8 

AM 

Batt  B 

8 

Station  3 

9 

FM  (Internal) 

CF  (Bn  1,  Bn  2) 

9 

Station  4 

10 

FM  (Internal) 

LBU  (LBU  Bn) 

10 

Station  5 

Wire 

• 

* 1 

Di v/Arty 

* 15 

Station  6 

2 

Bn  XO 

16 

Station  7 

3 

SI 

17 

Station  8 

* 4 

S2 

18 

Station  9 

5 

S3 

19 

Station  10 

6 

S4 

20 

Station  11 

* 7 

Comm  0. 

4 Wire 

* 8 

M.A.S.H. 

1 

2 

3 

4 


* Subscribers  connected  to  CTB. 
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Figure  17.  CCU  net  directory 


ecu  Operations.  In  this  segment  the  operators  were  required  to  perform 
ecu  operational  tasks.  The  segment  required  the  participation  of  all  five 
participants.  One  test  participant  operated  the  CCU,  and  the  other  four 
participants  manned  wire,  local,  and  radio  lines.  Each  participant  was 
provided  with  a script.  The  script  contained  a complete  list  of  task  opera- 
tions, indicating  the  duties  each  operator  was  to  perform.  Each  task 
represented  a discrete  event.  The  test  director  communicated  with  the 
operators  (via  an  intercom)  and  informed  them  what  task  was  to  be  performed 
next.  When  the  test  director  said,  "Task  N,  proceed",  the  operators  involved 
would  perform  their  required  tasks.  Response  time  and  accuracy  were  recorded 
on  the  checklist. 

CCU  Fault  Isolation  and  Card-Level  Maintenance.  For  this  segment  the 
operators  were  required  to  perform  the  following  two  operational  maintenance 
tasks: 

1.  CCU  Operational  Checkout 

2.  Card  Level  Maintenance. 

For  the  CCU  checkout,  the  operators  were  required  to  connect  one  wire,  one 
local,  and  one  radio  subscriber  to  different  nets  and  verify  that  the  CCU 
was  sending  out  a signal.  In  card  level  maintenance,  the  operators  performed 
several  keyboard  actuations  to  determine  whether  the  card  was  functioning. 

The  test  director  explained  the  procedures  for  performing  these  tasks  before 
testing  began.  Response  times  and  errors  were  recorded  on  the  checklist. 


Data  Reduction  Methods 

Means  and  standard  deviations  were  calculated,  and  reported  with  the 
tabulated  data.  Error  rates  were  reported,  as  well  as  a tabular  record  of 
errors  committed  during  one  test  segment.  Questionnaire  data  were  summarized 
in  narrative  form. 


HFE  TEST  RESULTS 


Performance  data  were  recorded  for  each  of  the  five  major  tasks.  This 
subsection  presents  the  results  of  each  HFE  test  separately  and  summarizes 
the  questionnaire  results. 


Performance  Data 

CTB-Subscriber  Connection.  In  this  segment  the  operators  connected  a 
total  of  thirty-five  circuits  to  the  CTB  (eleven  wire,  eight  radio,  and  six- 
teen local  circuits).  Table  23  presents  the  total  time  required  for  each 
operator  to  set  up  the  CTB.  On  the  average,  the  operators  took  13.2  minutes 
to  make  all  of  the  required  connections.  Operator  error  rate  for  this  task 
was  less  than  .01  (one  error  in  135  attempts). 
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TABLE  23 


Data  Summary  for  Test  Segment  1 (CTB  Set  Up) 


PARTICIPANT 

TIME 

(MINUTES) 

ERRORS 

1 

11.5 

None 

O 

£. 

9.4 

None 

3 

14.2 

Left  off  1 connection 

4 

16.6 

None 

5 

14.2 

None 

TOTAL 

65.9 

MEAN 

13.2 

STANDARD  DEVIATION 

2.8 

AVERAGE  TIME/CIRCUIT 

= 13.2/35 

= 0.4  MIN/CONNECTION 
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Equipment  Start  Up.  In  this  segment  the  operators  were  required  to 
perform  all  of  the  tasks  required  to  initialize  and  clear  the  CCU.  Table  24 
gives  the  total  time  each  operator  required  to  perform  the  tasks.  The  mean 
time  to  perform  equipment  start  up  procedures  was  40.1  seconds.  No  errors 
were  observed. 

Net  Set  Up.  In  this  segment,  the  operators  were  required  to  configure 
seven  nets^  Tne  number  of  subscribers  associated  with  each  of  the  nets  is 
shown  in  the  second  column  of  Table  25.  Table  25  also  presents  the  total 
time  required  by  each  operator  to  configure  the  nets.  The  average  times 
required  to  set  up  a net  and  to  connect  a circuit  to  a net  were  33.6  and  6.5 
seconds,  respectively.  No  errors  were  observed  in  setting  up  the  nets. 

CCU  Operations.  In  this  segment  the  operators  were  required  to  perform 
CCU  operational  tasks.  Table  26  contains  operator  response  times,  means, 
and  standard  deviations  for  each  task.  Tasks  which  involved  similar  oper- 
ations (i.e.,  set  up  links,  add/delete  subscribers  to  a net,  etc.)  were 
grouped  together  to  derive  mean  times  for  performing  those  operations  (Table 
27).  Table  28  lists  the  errors  committed  by  the  test  participants  while 
performing  this  segment. 

Descriptions  and  frequencies  of  the  most  prevalent  errors  are  shown  in 
Table  29.  The  cross-reference  and  reassignment  errors  (described  in  Section 
5.)  are  confounded  with  the  operator  disconnect  errors;  if  the  operator  failed 
to  push  the  OPR  and  DISC  buttons,  he  could  not  reassign  the  subscriber  to  his 
original  net.  Therefore,  these  three  errors  are  not  reported  separately.  A 
single  key  activation  error  was  scored  if  one  or  more  keys  were  activated 
inappropriately  or  in  the  wrong  order  during  any  single  task. 

CCU  Fault  Isolation  and  Card  Level  Maintenance.  In  this  segment  the 
operators  were  required  to  perform  operation  level  maintenance  on  the  CCU. 

The  tasks  were  performed  by  having  the  operators  press  specific  sequences  of 
keys  on  the  CCU  keyboard  to  "troubleshoot"  the  CCU.  Table  30  gives  the 
total  time  each  operator  required  to  perform  the  tasks.  The  average  times 
required  for  fault  isolation  and  card  checkout  were  11.93  and  5.75  seconds, 
respectively.  No  errors  were  observed  in  this  test  segment. 

In  the  fault  isolation  task,  the  operators  were  required  to  place  spec- 
ified subscribers  on  selected  nets  and  attempt  to  ring  the  subscriber.  If 
the  subscriber  was  connected  appropriately,  the  CCU  operator  would  hear  a 
ring  when  he  attempted  to  ring  the  subscriber.  In  the  card  checkout  test, 
the  operators  were  required  to  perform  a specified  sequence  of  key  actuations 
to  determine  whether  selected  cards  were  operable.  The  status  of  the  card 
being  tested  was  displayed  to  the  operator  on  the  error  display.  If  the 
card  was  good,  an  "80"  flashed;  and  if  the  card  was  to  be  replaced,  a "79" 
would  flash  in  the  error  display  window. 
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TABLE  24 


Data  Summary  for  Test  Segment  2 (Equipment  Start  Up) 


PARTICIPANT 

TIME 

(MINUTES) 

ERRORS 

1 

56.8 

None 

2 

21.0 

None 

3 

36.5 

None 

4 

32.0 

None 

5 

54.3 

None 

TOTAL 

200.7 

MEAN 

40.1 

STANDARD  DEVIATION 

15.2 
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TABLE  25 


Data  Summary  for  Test  Segment  3 (Net  Set  Up) 


Performance  Time  (Seconds) 
PARTICIPANT 


TOTAL 


NET 

SUBSCRIBERS/NET 

1 

2 

3 

4 

5 

6* 

1 

5 

38.4 

14.0 

33.3 

60.0 

31.4 

21.4 

2 

2 

14.6 

9.0 

23.2 

14.0 

23.0 

19.0 

3 

5 

22.9 

13.0 

35.5 

33.0 

34.3 

17.2 

4 

5 

28.6 

34.0 

38.6 

28.0 

31.7 

23.7 

5 

9 

51.3 

33.0 

52.6 

47.0 

44.8 

42.6 

6 

4 

25.6 

17.0 

37.9 

41.0 

73.1 

29.7 

7 

6 

33.0 

18.0 

43.7 

42.0 

46.6 

23.6 

TOTAL  36 

214.4 

138 

264.8 

276 

285 

177.1 

MEAN 

30.6 

19.7 

37.8 

39.3 

40.7 

25.3 

STANDARD  DEVIATION 

11.8 

9.9 

9.1 

14.0 

16.5 

8.6 

OVERALL  TIME/NET 

MEAN  = 33.6  seconds 


STANDARD  DEVIATION  = 

14.2 

seconds 

N 

35 

OVERALL  TIME/SUBSCRIBER 

MEAN 

6.5 

seconds 

STANDARD  DEVIATION  = 

2.6 

seconds 

N 

180 

♦Times  recorded  for  one  of  the  test  directors  — not  used  In  calculating 
average  net  times  or  standard  deviations. 
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TABLE  26 


Data  Summary  for  Test  Segment  4 (CCU  Operations)  I 


Performance  Time  (Seconds)  j 

PARTICIPANT 


TASK 

1 

2 

3 

4 

5 

MEAN 

STANDARD 

DEVIATION 

1. 

VERIFY 
CONNECTIONS 
a.  Radio  1 

31.2 

23.2 

25.7 

13.5 

37.0 

26.1 

8.9 

b.  Radio  5 

23.3 

48.5 

68.8 

21.3 

28.4 

38.0 

20.3 

c.  Wire  4 

11.6 

61.0 

10.7 

28.0 

33.3 

28.9 

20.5 

d.  Wire  7 

15.9 

39.7 

19.6 

30.1 

15.0 

24.1 

10.6 

e.  Wire  8 

35.8 

39.5 

17.2 

21.3 

20.6 

26.9 

10.1 

f.  Wire  1 

57.4 

122.0 

180.0 

20.5 

180.0 

102.0 

84.0 

g.  Local  2 

20.7 

15.3 

42.8 

18.1 

17.6 

23.0 

11.3 

h.  Local  5 

24.1 

42.0 

28.4 

21.1 

19.1 

26.9 

9.1 

1.  Local  15 

61.3 

40.7 

32.9 

22.3 

43.9 

40.2 

14.4 

TOTAL 

281.4 

432.0 

425.9 

185.9 

394.9 

37.3 

25.0 

2. 

VERIFY  NETS 

a.  Net  1 

5.7 

4.0 

9.0 

5.7 

5.0 

5.9 

1.9 

b.  Net  2 

4.6 

12.0 

10.0 

12.7 

4.0 

8.7 

4.1 

c.  Net  3 

13.0 

46.5 

8.0 

10.0 

7.0 

16.9 

16.7 

d.  Net  4 

6.7 

10.3 

11.0 

7.2 

8.0 

8.6 

1.9 

e.  Net  5 

8.6 

12.4 

12.0 

11.7 

9.0 

10.7 

1.8 

f.  Net  6 

5.6 

9.4 

10.0 

5.7 

7.0 

7.5 

2.1 

g.  Net  7 

6.0 

15.0 

9.4 

7.5 

7.0 

9.0 

3.6 

TOTAL 

50.1 

109.6 

69.4 

60.6 

47.0 

9.6 

3.5 

3. 

2-WAY  LINK 

183.4 

71.4 

438.2 

68.2 

268.6 

206.0 

154.5 

4. 

ADD  RADIO 

SUB  TO  NET 

59.4 

59.5 

85.8 

92.8 

131.4 

85.8 

29.7 

5. 

VERIFY  NETS 
a.  Net  1 

9.4 

5.9 

25.1 

5.8 

10.1 

11.2 

8.0 

b.  Net  4 

16.5 

7.8 

50.2 

6.6 

16.0 

19.4 

17.8 

TOTAL 

25.8 

13.6 

75.3 

12.4 

26.1 

15.3 

13.7 

6. 

3-WAY  LINK 

144.3 

76.4 

203.9 

213.6 

60.0 

139.6 

70.7 

7. 

2-WAY  LINK 

119.5 

44.3 

119.2 

108.4 

30.7 

84.4 

43.3 

8. 

2-WAY  LINK 

34.5 

28.3 

132.2 

99.5 

48.2 

68.5 

45.3 

9. 

ANS  RING  IN 

21.1 

30.9 

45.9 

19.4 

20.0 

27.5 

11.3 

(continued) 
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PARTICIPANT 


TASK 

1 

2 

3 

4 

5 

MEAN 

STANDARD 

DEVIATION 

10. 

ADD  SUB  TO 

NET 

24.2 

24.9 

33.8 

108.4 

31.5 

42.6 

37.1 

11. 

VERIFY  NETS 
a.  Net  1 

6.8 

3.9 

49.3 

9.3 

21.4 

18.2 

18.7 

6.  Net  3 

21.0 

9.1 

28.5 

10.9 

10.6 

16.0 

8.5 

TOTAL 

27.8 

13.0 

117.9 

20.3 

32.0 

17.1 

13.7 

12. 

MONITOR  NET 
TRAFFIC 

7.5 

31.9 

20.0 

18.5 

17.9 

19.2 

8.7 

13. 

DISC  LINK 

268.9 

35.0 

23.6 

25.9 

83.8 

87.4 

104.3 

14. 

RECONFIGURE 
NETS 
a.  Net  1 

14.7 

14.7 

49.6 

29.8 

15.3 

24.3 

15.2 

b.  Net  2 

15.3 

23.8 

25.1 

41.2 

24.0 

25.9 

9.4 

c.  Net  3 

23.6 

25.1 

28.7 

43.0 

19.0 

28.9 

11.2 

d.  Net  4 

35.3 

38.2 

25.1 

63.9 

33.7 

39.2 

14.6 

e.  Net  5 

9.1 

46.8 

38.2 

49.6 

18.9 

32.5 

17.8 

f.  Net  6 

11.0 

18.7 

25.0 

35.7 

25.4 

23.2 

9.1 

g.  Net  7 

23.6 

21.5 

25.0 

31.9 

21.0 

24.6 

4.4 

TOTAL 

132.6 

183.8 

216.7 

300.1 

157.3 

28.4 

9.7 

15. 

ADD  2 SUBS 

J 

TO  NET  1 

28.5 

10.6 

105.7 

22.0 

50.3 

43.4 

37.7 

16. 

VERIFY  NETS 

5.0 

6.4 

7.0 

6.8 

3.2 

5.7 

1.6 

17. 

2-UAY  LINK 

79.7 

74.0 

136.1 

197.4 

151.4 

127.7 

51.7 

18. 

2-IVAY  LINK 

83.9 

19.1 

50.1 

63.3 

45.0 

52.3 

23.9 

19. 

ADO  SUB  TO 

40.3 

48.5 

50.5 

64.2 

54.4 

51.6 

8.8 

NET 

20. 

SNITCH  DDT 
FROM  1 NET 

TO  ANOTHER 

56.7 

33.9 

97.1 

55.8 

43.2 

57.3 

?4.2 

21. 

VERIFY  NETS 
a.  Net  I 

135.0 

39.3 

28.8 

21.2 

60.4 

56.9 

46.1 

b.  Net  6 

25.0 

6.1 

36.3 

24.0 

171.0 

52.5 

67.1 

TOTAL 

160.0 

45.3 

65.0 

45.3 

231.4 

54.7 

54.3 

(continued) 
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PARTICIPANT 


TASK 

1 

2 

3 

4 

5 

MEAN 

STANOARD 

22. 

DISC  SUB 

FROH  NET 

27.5 

28.9 

25.1 

46.6 

67.5 

39.1 

DEVIATION 

18.0 

23. 

VERIFY 

CHANGE 

8.7 

14.3 

10.8 

8.6 

8.0 

10.1 

2.6 

24. 

2-UAY  LINK 

127.0 

24.0 

208.9 

120.9 

39.0 

104.0 

74.9 

25. 

LOCAL  DOWN 
CHANGE  CTB 
a.  Receive 
Order 

59.4 

57.7 

60.8 

57.6 

130.1 

73.1 

31.9 

b.  Set  Up 

CTB 

52.6 

25.7 

35.1 

30.0 

29.0 

34.5 

10.7 

c.  Ring  Up 

LCL 

35.1 

59.7 

10.0 

14.9 

10.6 

26.1 

21.4 

TOTAL 

147.1 

143.1 

105.9 

102.5 

169.7 

44.6 

30.1 

26. 

MAKE  UP  NEW 
NET 

183.2 

130.1 

129.9 

354.3 

131.5 

185.8 

96.9 

27. 

RECONFIGURE 
NETS 
a.  Net  1 

11.0 

35.3 

90.7 

33.8 

14.9 

37.1 

31.9 

b.  Net  2 

25.6 

15.2 

29.8 

28.7 

30.6 

26.0 

6.3 

c.  Net  3 

13.9 

13.8 

16.6 

14.1 

28.1 

17.3 

6.2 

d.  Net  4 

26.0 

20.6 

28.4 

31.9 

25.4 

25.1 

6.8 

e.  Net  5 

37.4 

39.3 

51.6 

57.9 

48.6 

43.2 

14.7 

f.  Net  6 

19.4 

16.4 

20.7 

27.1 

19.1 

20.5 

3.6 

g.  Net  7 

23.6 

19.9 

61.5 

46.9 

24.0 

35.2 

18.2 

h.  Net  8 

43.0 

36.4 

48.3 

64.7 

26.0 

43.7 

14.4 

TOTAL 

183.2 

197.0 

347.6 

345.1 

226.8 

31.0 

20.6 

(concluded) 
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TABLE  27 


Summary  of  CCU  Operational  Tasks 


TASK 

OPERATION  TIME 

j 

MEAN 

STANDARD 

DEVIATION 

N j 

] 

35 

1. 

Set  Up  a Link 

111.8  Seconds 

51.8 

Seconds 

2. 

Disconnect  a Link 

87.4  Seconds 

104.3 

Seconds 

5 

3. 

Verify  a Net 

16.7  Seconds 

8.1 

Seconds 

80 

4. 

Add/Delete  a 
Subscriber  From 
a Net 

52.5  Seconds 

19.2 

Seconds 

25 

1 

5. 

Reconfigure  a Net 
After  CCU  Power 
Failure 

29.9  Seconds 

8.2 

Seconds 

75 

j 

6. 

Reallocate  DDT  to 
a Different  Net 

57.3  Seconds 

24.2 

Seconds 

5 

7. 

Answer  a Ring-In 

From  a Subscriber 

27.5  Seconds 

6.8 

Seconds 

5 

8. 

Switch  a Subscriber 
to  a New  Circuit 
(Req.  Operator  to 
Change  Connection 
in  the  CTB). 

222.8  Seconds 

125.4 

Seconds 

5 

9. 

Verify  a Subscriber 
is  Connected  to  CCU 

37.3  Seconds 

25.0 

Seconds 

45 

10. 

Establish  a New 

Net 

185.6  Seconds 

96.9 

Seconds 

5 

11. 

Monitor  Net  Traffic 

19.2  Seconds 

8.7 

Seconds 

5 
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TABLE  28 

Error  Summary  for  Test  Segment  4 


PARTICIPANT 


TASK 

1 

2 

3 

4 

5 

1. 

VERIFY 

CONNECTIONS 

, Failure  to  Actuate 
KEY  ON 

Headset 

Inappropriately 

Placed 

Key  Actuation 

Errors 

No  KEY  ON 

Key  Actuation 
Errors 

OPR  DISC 

Error 

No  KEY  ON 

Failure  to  OPR 
DISC 

Key  Errors 

Headset 

Inappropriately 

Placed 

No  KEY  ON 

Key  Errors 

OPR  DISC  Error 
Headset 

Inappropriately 

Placed 

z. 

VERIFT  NETS 

Left  Subscribers 
Off  Nets 

Key  Errors 

Left  Subscriber 
Off  A Net 

Kept  Subscribers 

Off  Nets 

3. 

2-WAY  LINK 

Errors  In  Setting 

Up  Link 

Key  Errors, 

OPR  DISC 

Error 

Key  Errors, 

OPR  DISC 

Error 

Errors  in  Setting 

Up  Link 

Errors  in  Setting 

Up  Link 

4. 

ADD  SUB  TO 

NET 

OPR  DISC 

Error 

OPR  DISC 

Error 

Problem  Adding 

Sub.  to  Net, 

OPR  DISC 

Error 

OPR  DISC 

Error 

5. 

VERIFY  NETS 

Sub. Left  Off  Net 

Sub. Left  Off  Net 

Sub. Left  Off  Net 

Sub. Left  Off  Net 

6. 

3-WAY  LINK 

Key  Errors 

OPR  DISC 

Error 

Problem  Setting 

Up  Link 

Problem  Setting 

Up  Link 

7. 

2-WAY  LINK 

1 

[__ 

Key  Errors 

Problem  Setting 
Up  Link 

Key  Errors 

Problem  Setting 

Up  Link 

OPR  DISC 

Error 

OPR  DISC  Error 

8. 

2-WAY  LINK 

OPR  DISC  ERROR 

Problem  Setting 

Up  Link  - Lost 
Subscriber 

Problem  Setting 

Up  Link  - Lost 
Subscriber 

9. 

ANSWER  RING  IN 

OPR  DISC  Error 

10. 

ADD  SUB. TO  NET 

Key  Errors 

OPR  DISC  ERROR 

Key  Errors 

OPR  DISC  Error 

OPR  DISC  ERROR 

11. 

VERIFY  NETS 

Subscribers 

Missing 

Key  Errors 

Subscriber 

Missing 

12. 

MONITOR  NET 
TRAFFIC 

13. 

DISCONNECT 

LINK 

OPR  DISC  Error 

Key  Errors 

Key  Errors 

OPR  DISC  Error 

14. 

RE-ESTABLISH 
NETS  AFTER 
POWER  FAILURE 

Key  Errors 

Left  Off 
Subscribers 

Left  Off 
Subscribers 

Left  Off 

Subscribers 

15. 

ADO  2 

SUBSCRIBERS 

TO  A NET 

OPR  DISC  Error 

Key  Errors 

OPR  DISC  Error 

Key  Errors 
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(coiitinued) 


PARTICIPANT 


TASK 

1 

2 

3 

4 

5 

16.  VERIFY  NETS 

17.  2-WAY  LINK 

OPR  DISC  Error 

No  KEY  ON 

Key  Errors 

No  KEY  ON 

Problem  Setting 

Un  Link 

No  KEY  ON 

Key  Errors 

No  KEY  ON 

18.  2-WAY  LINK 

OPR  DISC  Error 

Key  Errors 

Key  Errors 

Key  Errors 

19.  ADD  SUBSCRIBER 
TO  A NET 

Key  Errors 

Key  Errors 

Key  Errors 

2D.  SWITCH  DDT 

FROM  1 NET 

TO  ANOTHER 

OPR  DISC  Error 

OPR  DISC  Error 

Key  Errors 

OPR  DISC  Error 

OPR  DISC  Error 

OPR  DISC  Error 

21.  VERIFY  NETS 

Missing 

Subscribers 

Missing  Subs. 
Key  Errors 

Missing  Subs. 

Key  Errors 

Missing  Subs. 

Missing  Subs. 

22.  DISCONNECT  A 
SUBSCRIBER 

FROM  A NET 

Key  Errors 

23.  VERIFY  NETS 

24.  2-WAY  link 

1 Key  Errors 

OPR  DISC  Error 

Key  Errors 

25.  LOCAL  DOWN 
CHANGE  CTB 

Key  Errors 

26.  MAKE  UP  A 

NEW  NET 

OPR  DISC  Error 
Missing 
Subscribers 

Key  Errors 

Key  Errors 

Missing 

Subscribers 

27.  RE-ESTABLISH 
NETS  AFTER 

POWER  FAILURE 

' Key  Errors 

Left  Off  1 
Subscriber 

Key  Errors 

Left  Off  1 

Subscriber 

(concluded) 
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TABLE  29 


Frequencies  and  Descriptions  of  Major  Errors 


ERROR  DESCRIPTION  FREQUENCY  ERROR  RATE 


1.  KEY-ON  Error  (Failure  to  8 Errors  in  10  Attempts  80% 

actuate  KEY-ON  button  when 
attempting  to  communicate 
with  another  CCU) 


2.  Headset  Connection  Error  3 Errors  in  5 Attempts  60% 

(Operator  connected  headset 
connector  into  inappropriate 
jack) 


3.  Opr. -Disc.  Error  (Operator's  30  Errors  in  90  Attempts  33% 

failure  to  actuate  the 
OPERATOR  and  DISCONNECT  keys 
after  a subscriber  has 
terminated  communication  with 
the  operator) 


4.  Key-Actuation  Errors 

Key  Errors  Occurred  in 

28% 

(Operator  errors  in 

38  of  the  135  Tasks 

pressing  incorrect  keys. 

Performed. 

incorrect  sequencing  of 

135  = 5 operators  X 

key  actuations,  etc.) 

27  tasks 

5.  Link  Error  (Operator 

difficulty  in  establishing 

9 Errors  in  35  Attempts 

26% 

links,  independent  of  the 
operator  disconnect  error) 
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TABLE  30 


Data  Summary  for  Test  Segment  5 
(Fault  Isolation  and  Card  Check-Out) 


Performance  Time  (Seconds) 
PARTICIPANT 


TASK 

1 

2 

3 

4 

5 

MEAN 

STANDARD 

DEVIATION 

FAULT  ISOLATION 

WIRE 

17.7 

7.0 

12.0 

13.0 

13.4 

12.6 

3.8 

LOCAL 

22.1 

5.0 

9.0 

8.0 

15.6 

11.9 

6.9 

RADIO 

16.8 

3.0 

9.0 

8.0 

19.3 

11.2 

6.7 

TOTAL 


11.9  5.6 


CARD  CHECKOUT 


WIRE 

9.0 

6.0 

3.3 

5.0 

5.1 

5.7 

LOCAL 

6.7 

5.7 

4.3 

3.7 

8.8 

4.6 

RADIO 

7.0 

4.7 

4.6 

3.7 

8.6 

5.7 

total  5.7  1.9 
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Questionnaire  Data 

The  questionnaire  was  administered  to  the  operators  after  they  had 
completed  testing.  The  main  findings  obtained  from  the  questionnaire  are 
suttmarized  below: 

Training  and  Practice.  All  of  the  operators  felt  that  more  practice  on 
real  equipment  and  more  on-the-job  training  are  required.  The  operators 
reported  that  more  "hands-on"  experience  would  have  increased  their  operating 
efficiency  (less  time  to  complete  tasks  and  a lower  error  rate). 

Task  Interest.  All  of  the  operators  found  the  task  of  operating  the 
ecu  to  be  interesting.  Most  respondents  felt  that  an  eight-hour  duty  cycle 
should  be  the  maximum  duty  time.  The  operators  were  not  experienced  enough 
to  estimate  time-to-fatigue  or  boredom  in  operating  the  CCU. 

ecu  Operating  Procedures.  The  operators  felt  that  most  of  the  problems 
they  had  operating  the  CCU  could  have  been  alleviated  with  more  hands-on 
experience.  No  respondent  reported  a problem  in  determining  when  a task 
was  started.  The  main  operating  problems  reported  by  the  participants 
included: 

1.  Updating  subscriber  forms 

2.  Determining  whether  pushbuttons  were  actuated 

3.  Establishing  and  operating  nets 

4.  Establishing  links. 

Man/Machine  Interface.  Most  of  the  operators  felt  that  the  CCU  van  was 
too  crowded  and  that  the  CCU  console  should  be  placed  outside  of  the  van. 

The  only  interface  problem  reported  was  determining  whether  a pushbutton  was 
actuated.  Most  of  the  operators  reported  they  were  comfortable,  and  they 
all  stated  that  they  could  easily  reach  the  CCU  control  panel. 

Working  Environment.  The  main  item  of  agreement  among  operators  was 
that  the  CCU  van  was  too  crowded.  The  only  other  item  checked  by  more  than 
one  operator  was  that  the  noise  produced  by  the  hardware  in  the  van  inter- 
fered with  their  ability  to  hear.  Other  items  checked  were  not  internally 
consistent  (one  operator  reported  that  it  was  too  dark  in  the  van,  while 
another  operator  reported  that  it  was  too  bright,  etc.). 


End  of  Test  Session  Interview 

Each  operator  was  interviewed  after  he  had  completed  his  testing.  The 
interviews  were  conducted  to  obtain  information  about  the  participants' 
feelings  and  attitudes  toward  their  training,  operational  procedures,  and 
testing.  The  data  supplemented  the  data  that  had  been  acquired  from  ques- 
tionnaires. The  principal  findings  obtained  from  the  interviews  and  not 
reported  elsewhere  are  listed  next: 


122 


1.  All  operators  stated  that  more  leg  room  should  be  provided. 

2.  All  of  the  operators  reported  that  they  needed  more  adequate 
writing  space. 

3.  The  displayed  data  were  legible,  and  the  operators  did  not 
find  the  displayed  information  confusing. 

4.  All  operators  stated  that,  without  the  subscribers'  directories, 
they  would  have  had  extreme  difficulty  in  remembering  the 
subscribers'  locations. 

5.  None  of  the  operators  were  tired  when  they  were  tested. 

6.  All  felt  that  their  training  was  inadequate.  They  all  said 
that  additional  hands-on  experience/OJT  is  required.  All 
stated  that  scenario-based  instruction  would  be  extremely 
useful  during  training. 


HUMAN  PERFORMANCE  PROBLEMS/ ERRORS 

This  section  discusses  the  major  problems  and  errors  observed  during 
the  test.  Descriptions,  frequencies,  consequences,  causes,  and  alternative 
solutions  or  actions  are  provided  for  each  error.  The  errors  are  discussed 
in  order  of  importance  and  rated  in  terms  of  their  consequences  on  system 
performance. 


Operator  Disconnect  Error 

Error  Description  and  Frequency.  This  was  one  of  the  most  prevalent 
errors  observed.  Thirty-three  percent  of  the  time  (thirty  out  of  ninety 
occurrences)  operators  failed  to  activate  the  operator  (OPR)  and  disconnect 
(DISC)  keys  after  terminating  conversations  with  subscribers.  A subscriber 
is  automatically  placed  on  the  operator  circuit  when  the  CCU  operator 
answers  the  subscriber's  ring-in.  The  CCU  operator  must  manually  remove 
the  subscriber  from  this  circuit  by  activating  the  OPR  and  DISC  keys  before 
he  can  place  the  subscriber  on  a net  or  a link. 

Error  Consequence.  The  effect  of  this  error  is  that  the  subscriber  is 
effectively  "lost"  from  the  communcation  nets  until  the  CCU  operator  "finds" 
the  subscriber  or  until  the  subscriber  rings  in  again.  In  the  meantime,  the 
subscriber  will  miss  any  communications  occurring  on  the  net  or  link  to 
which  he  should  have  been  connected. 

Error  Cause.  Inappropriate  allocation  of  function  was  the  principal 
factor  in  producing  this  error.  The  operator  is  required  to  press  the  OPR 
and  DISC  buttons  to  disconnect  a subscriber  from  the  operator  circuit  before 
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connecting  him  to  a net  or  link.  This  task  is  more  appropriately  allocated 
to  the  machine. 


Alternative  Solutions  or  Actions.  This  error  could  be  eliminated  by 
modifying  the  software,  or  reduced  by  training.  Since  the  CCU  is  primarily 
controlled  by  software,  a possible  solution  to  the  problem  would  be  control- 
ling the  disconnect  function  through  software.  The  software  might  work  in 
the  following  manner.  After  the  subscriber  had  completed  his  call,  he 
could  ring  off.  This  act  could  signal  the  computer  to  disconnect  him  from 
the  operator  circuit  and  place  him  back  on  the  circuit  he  was  on  before  he 
called  the  CCU  operator.  If  the  subscriber  asked  to  be  placed  on  another 
circuit,  he  would  not  ring  off;  the  CCU  operator  would  place  him  onto  the 
other  circuit.  This  act  would  automatically  remove  the  subscriber  from  the 
operator  circuit  and  place  him  on  the  desired  circuit.  Alternative  software 
modifications  could  be  used;  however,  the  impact  of  such  modifications  on 
total  system  operation  should  be  investigated,  since  allocating  this  active 
disconnect  function  to  software  may  have  other  consequences  that  are  not 
considered  here. 

A general  design  philosophy  for  the  CCU  should  be  "active  connect  and 
passive  disconnect."  With  this  procedure,  an  operator  would  have  to  take 
positive  action  to  answer  or  connect  a subscriber  but  not  to  disconnect  him 
or  to  place  him  at  his  "home"  location. 

With  the  present  CCU  design,  training  that  stresses  operator  discon- 
nection should  reduce  the  operator  disconnect  error.  However,  even  the  well- 
practiced  operator  experienced  this  problem,  so  training  alone  will  probably 
not  totally  eliminate  this  error.  Increased  training  will  make  the  operator 
more  aware  of  the  possibility  of  the  error.  Therefore,  the  trained  operator 
would  be  more  likely  to  look  for  a "lost"  subscriber  on  the  operator  circuit, 
so  he  could  remedy  his  errors  rapidly. 


Key  Actuation  Errors 

Error  Description  and  Frequency.  In  twenty-eight  percent  of  the  tasks, 
the  operator  made  key  actuation  errors.  Key  actuation  errors  included: 

1.  Pressing  an  inappropriate  key  (e.g.,  an  operator  was  required 
to  actuate  the  NET  key  but  inadvertently  actuated  the  LINK  key) 

2.  Actuating  keys  out  of  sequence 

3.  Depressing  a key  without  actuating  it  (i.e.,  applying  a force 
that  is  insufficient  to  actuate  the  key). 

Error  Consequences.  The  consequences  of  this  error  ranged  from  in- 
creasing  the  time  required  to  perform  tasks  to  assigning  the  subscriber  to 
an  undesired  net  or  link  or  removing  him  from  the  desired  net  or  link.  If 
an  operator  detected  the  error,  he  could  clear  the  keyboard  and  retype  the 
desired  command.  But,  if  the  operator  did  not  detect  the  error,  subscribers 
were  assigned  improperly.  Misassigning  a subscriber  means  that  he  could  miss 
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important  cottmuni cations.  This  error  does  not  affect  the  subscribers' 
ability  to  communicate  with  the  CCU  operator. 

Error  Causes.  Improper  key  actuation  was  caused  primarily  by  the 
following  three  factors: 

1.  Inadequate  tactile  sensory  feedback 

2.  Excessive  displacement  of  the  display  from  the  keyboard 

3.  Close  key  spacing. 

The  present  CCU  keyboard  uses  rubberized  keys.  The  sensory  information 
provided  by  depressing  the  keys  was,  in  some  cases,  inadequate.  To  determine 
whether  he  had  depressed  the  key,  the  operator  had  to  look  at  the  status 
display,  which  caused  an  unnecessary  delay.  The  status  display,  like  a 
typewriter,  is  outside  of  the  operator's  normal  viewing  area  when  he  is 
operating  the  keyboard. 

Another  keyboard  design  feature  that  contributed  to  error  was  the  close 
spacing  between  keys.  Center-to-center  key  spacings  varied  from  .06"  to 
.13".  MIL-STD-1472B  (36)  specifies  that  the  center-to-center  spacing  should 
not  be  less  than  .635"  and  minimum  key  separation  should  be  .25".  Closely 
spaced  keys  contributed  to  the  operators'  errors  in  depressing  adjacent 
keys  incorrectly.  Most  of  these  errors  were  detected  and  corrected. 

Alternative  Solutions/Actions.  Redesigning  the  keys  so  that  they  provide 
definite  tactual  feedback  when  operated  would  eliminate  the  operators' 
problem  of  determining  whether  a key  had  been  actuated.  An  approach  which 
has  been  successfully  used  on  other  sealed,  flat-plate  switch  keyboards  is 
mounting  a relay  on  a sounding  plate  so  it  produces  an  auditory  and  tactile 
signal  whenever  a key  is  actuated. 

Spacing  of  the  keys  farther  apart,  so  they  conform  to  military  specifi- 
cations, could  help  eliminate  the  operators'  problem  of  inadvertently 
actuating  inappropriate  keys.  During  testing,  it  was  also  observed  that  the 
legends  on  some  of  the  rubberized  keys  were  wearing  off.  It  is  recommended 
that  the  keys  be  redesigned  to  prevent  the  errors  that  will  probably  occur 
when  the  labels  wear  off. 


Link  Errors 

Error  Description  and  Frequency.  During  CCU  operations  two  or  more 
subscribers  can  be  placed  togetner  on  a common  comnunication  circuit  called 
a link.  A link  is  similar  to  a net,  except  that  a Digital  Data  Terminal  (DDT) 
can  be  included  on  a net  but  not  on  a link.  Link  errors  were  observed  during 
twenty-six  percent  of  the  trials  that  involved  link  operations.  Errors 
observed  in  setting  up  links  included: 
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1.  Attempting  to  connect  subscribers  on  the  same  net  before  linking 

them  I 

2.  Failing  to  connect  all  required  subscribers  to  the  link  I 

3.  Linking  one  subscriber  at  a time  and  communicating  with  him 

before  connecting  other  subscribers,  instead  of  linking  all  i 

required  subscribers  and  then  communicating  to  them  all  by  ' 

talking  on  the  link.  ^ 

The  only  procedural  difference  between  establishing  nets  and  links  is  in  > 

pressing  the  NET  or  LINK  key.  The  operators'  difficulty  in  establishing 
links  is  indicated  by  the  significantly  longer  time  required  to  set  up  links 
(111.8  seconds)  as  compared  with  setting  up  nets  (33.6  seconds). 

Another  error  associated  with  link  operations  occurred  when  an  operator 
correctly  disconnected  all  subscribers  from  the  link  by  pushing  the  disconnect 
(DISC)  key  three  times.  This  procedure  places  all  the  previously  linked 
subscribers  on  the  unassigned  subscribers  net  (Net  9).  On  several  occasions, 
the  operators  forgot  to  reassign  the  subscribers  to  their  original  circuits. 

Error  Consequences.  The  main  consequences  of  link  errors  ranged  from 
delays  in  transmitting  desired  communications  to  the  possibility  of  missing 
critical  transmissions.  The  former  results  from  the  excessive  time  required 
to  set  up  a link,  and  the  latter  because  disconnecting  the  subscriber  from 
a link  also  removes  him  from  the  communication  channel  he  desires. 

Error  Causes.  The  main  factor  that  contributed  to  the  operators' 
difficulty  in  establishing  links  was  that  training  did  not  address  the 
distinction  between  nets  and  links.  The  major  factor  which  contributed  to 
errors  in  disconnecting  links  was  the  equipment  design,  which  requires 
actively  reassigning  the  subscriber  to  his  original  circuit.  This  is  another 
case  where  a design  philosophy  of  "active  connect  and  passive  disconnect" 
would  eliminate  operator  errors. 

Alternative  Solutions/ Actions.  To  reduce  the  operators'  difficulty  in 
establishing  links,  the  distinction  between  net  and  link  functions  and 
procedures  should  be  covered  in  training.  Procedures  for  connecting  and 
disconnecting  subscribers  to  a link  are  nearly  identical  with  the  procedures 
for  net  circuits.  However,  if  the  similarities  are  not  highlighted  during 
training,  the  operators  become  confused  and  behave  as  though  link  operations 
require  special  procedures. 

The  link  disconnect  error  can  be  eliminated  by  allocating  to  the 
equipment  the  function  of  reinstating  subscribers  to  their  home  locations 
when  a link  is  disconnected.  The  impact  of  redesigning  the  software  on 
total  system  performance  should  be  investigated  before  making  this  change. 

If  the  software  cannot  be  restructured,  the  operators  should  receive  more 
instruction  or  on-the-job  training  (OJT)  which  highlights  the  procedure 
for  reconnecting  subscribers  after  disconnecting  them  from  a link. 
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KEY  ON  Error 


Error  Description  and  Frequency.  For  one  CCU  to  communicate  with  another, 
both  operators  must  activate  their  respective  KEY  ON  buttons.  Eighty  percent 
(8  of  10)  of  the  time,  the  test  operators  failed  to  activate  the  KEY  ON 
button  when  connecting  a CCU  to  a net. 

Error  Consequence.  The  consequence  of  this  error  is  that  CCUs  cannot 
communicate  until  both  operators  actuate  their  KEY  ON  buttons.  Also,  sub- 
stantial periods  of  time  were  required  to  rectify  the  error.  On  the  average, 
it  takes  much  longer  to  verify  connections  with  a CCU  subscriber  (102.0 
seconds)  than  it  does  to  verify  connections  with  all  other  subscribers  (30.2 
seconds) . 

Error  Causes.  Failure  to  actuate  the  KEY  ON  button  was  caused  primarily 
by  the  following  three  features: 

1.  The  CCU  operator's  training  course  did  not  describe  the  KEY  ON 
function 

2.  The  KEY  ON  function  is  used  only  when  connecting  another  CCU  to 
a net 

3.  This  function  was  allocated  to  the  operator  rather  than  to  the 
hardware 

While  the  KEY  ON  function  was  covered  in  the  pre-test  refresher  training,  it 
was  not  covered  in  the  participants'  previous  training.  This,  together  with 
the  fact  that  the  KEY  ON  function  is  seldom  used,  contributed  to  the  opera- 
tor's forgetting  to  actuate  the  KEY  ON  button.  The  immediate  cause  of  the 
error  was  that  the  CCU  operator  had  to  actuate  the  KEY  ON  button  before 
communicating  with  another  CCU,  but  he  could  connect  any  other  subscriber  to 
a net  simply  by  pushing  the  device's  name  button  (e.g.,  the  RAD  [radio] 
button  followed  by  the  CONN  [connect]  button). 

Alternative  Solutions/Actions.  Allocating  the  KEY  ON  function  to  soft- 
ware will  eliminate  this  error.  As  with  other  recommended  software  changes, 
the  impact  of  redesigning  the  software  to  accomplish  this  function  should  be 
investigated  to  determine  its  overall  effect  on  total  system  performance. 

If  it  is  determined  that  allocating  the  function  to  software  is  not  feasible, 
increased  operator  training  a.id  job  aids  should  be  developed  to  reduce  the 
frequency  of  this  error.  For  example,  the  form  used  to  record  subscriber 
locations  could  include  a written  reminder  to  the  operator  to  be  sure  to 
acutate  the  KEY  ON  button  when  connecting  a CCU  to  a net. 


Headset  Connection  Error 

Error  Description  and  Frequency.  Two  headset  connections  have  been 
provided  on  the  CCU.  Plugging  the  headset  into  the  upper  connector  allows 
the  operator  to  communicate  with  the  nets  configured  by  the  CCU.  Connecting 
it  to  the  lower  jack  gives  the  operator  the  same  capability,  plus  the  option 
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of  answering  calls  from  local  subscribers.  Sixty  percent  of  the  time,  the 
test  operators  plugged  the  headset  connections  into  the  wrong  (upper)  jack. 

Error  Consequence.  The  result  of  this  error  was  that  the  operators 
could  not  communicate  with  subscribers  until  they  connected  the  headset  to 
the  lower  jack. 

Error  Causes.  The  primary  cause  of  this  error  was  that  the  headset 
jacks  were  not  labeled.  Also,  most  operators  were  unaware  of  the  differences 
between  the  two  headset  connectors,  which  added  to  their  confusion. 

Alternative  Solutions/Actions.  Placing  a written  description  of  its 
functions  next  to  each  jack  should  nearly  eliminate  the  error,  and  it  would 
aid  the  operator  in  correcting  any  errors  that  do  occur.  The  test  directors 
were  unable  to  determine  whether  two  jacks  were  required.  It  was  assumed 
that  since  two  jacks  were  installed,  both  are  required.  If  it  turns  out  that 
one  jack  is  sufficient,  the  problem  could  be  solved  by  removing  the  unneeded 
jack. 


INCOMPATIBILITIES  AMONG  HUMAN  PERFORMANCE  AND  EQUIPMENT 


Task  Group  Interference 

The  only  task  group  interference  observed  during  the  test  was  between 
the  ecu  and  ACC  operators.  The  ACC  operator  station  is  immediately  adjacent 
to  and  in  front  of  the  CCU  station.  For  the  CCU  operator  either  to  reach 
or  leave  his  station,  the  ACC  operator  has  to  stand  up  and  move  his  chair. 
The  CCU  operator's  personal  space  (elbow  room)  is  limited  by  the  van  configu 
ration  and  by  the  presence  of  the  ACC  operator.  The  CCU  operator  does  not 
have  sufficient  space  to  record  information  or  to  stretch  out.  Figures  18, 
19,  and  20  illustrate  task  group  interference  and  show  the  inadequate  space 
provided  for  recording  data.  Figure  18  illustrates  the  relative  positions 
of  the  ACC  and  CCU  operators.  The  actual  van  floor  space  is  indicated  by 
the  striped  floor  area  in  the  figure.  As  may  be  seen  in  Figures  19  and  20, 
the  CCU  operator  does  not  have  sufficient  space  for  a work/writing  platform. 
Figure  19  shows  a CCU  operator  seated  in  front  of  the  CCU.  In  a normal  van 
the  back  of  his  chair  would  butt  up  against  an  equipment  rack.  Figure  20 
shows  the  CCU  and  its  associated  connections. 


Equipment  Incompatibilities 

No  equipment  incompatibilities  were  observed. 


OBSERVED  SAFETY  HAZARDS 

No  safety  hazards  were  observed  during  testing. 
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IMPACT  OF  HUMAN  PERFORMANCE  ON  ATTAINMENT  OF  SYSTEM  PERFORMANCE  GOALS 


System  Performance  Goals 

TACFIRE  system  performance  standards  are  specified  in  terms  of  the 
system's  response  times  to  complete  26  field  artillery  missions.  These 
mission  response  times  include  data  processing  and  transmission  times; 
however,  response  times  for  personnel  tasks  are  not  included.  In  particular, 
the  defined  mission  times  assume  that  all  cotranunication  lines  are  pre- 
established,  implying  either  that  the  CCU  operator's  tasks  are  completed 
prior  to  any  artillery  mission  or  that  the  CCU  operator  completes  his  tasks 
during  a mission,  with  no  errors,  in  an  infinitely  short  time  period. 

Clearly,  the  last  alternative  is  unreasonable.  It  is  also  unreasonable  to 
assume  that  all  communication  links  will  be  in  place  before  all  possible 
missions. 

In  the  absence  of  defined  CCU  operator  performance  standards,  a base- 
line task  was  selected  to  demonstrate  how  human  performance  impacts  on 
system  response  times.  The  selected  baseline  task  was  "set  up  a net", 
identified  as  subtasks  3.1  and  3.2  in  Table  18.  This  subtask  was  completed 
in  a minimally  short  time  (mean  = 33.6  seconds/net),  the  variance  among  test 
participants  was  low  (standard  deviation  = 14.2  seconds/net),  and  no  errors 
were  observed  in  setting  up  the  nets.  Therefore,  this  subtask  represented 
the  minimum  time  and  error  rate  that  could  be  expected  for  most  CCU 
operations. 


Human  Performance  Standards 

All  human  performance  tasks  identified  for  the  CCU  operator  have  been 
shown  by  this  HFE  test  to  be  feasible.  However,  the  performance  standardb 
for  TACFIRE  explicitly  exclude  the  human  performance  component.  Thus,  it 
was  not  possible  to  assess  whether  any  human  performance  standards  can  be 
met.  As  indicated  in  the  following  paragraphs,  human  performance  has  been 
compared  with  mission  times  assumed  for  TACFIRE,  even  though  those  mission 
times  specifically  excluded  the  human  performance  component.  According  to 
this  comparison,  human  performance  can  be  expected  to  exceed  significantly 
the  time  assumed  for  mission  accomplishment. 


Impact  of  Problems  and  Errors 

These  HFE  test  data  indicate  that  a CCU  operator  can  be  expected  to 
establish  the  initial  communications  between  subscribers  of  a TACFIRE  unit 
in  a timely  and  relatively  error-free  manner.  Test  operators  attached  the 
subscriber  lines  to  the  CTB,  initialized  and  cleared  the  CCU,  and  established 
all  initial  communication  nets  with  only  one  error  in  more  than  400  opera- 
tions. However,  a CCU  operator  can  be  expected  to  make  a considerable  number 
of  errors  during  CCU  operations  when  subscribers  are  using  the  communication 
nets.  These  errors  range  from  minor  key  actuation  errors,  which  are  detected 
and  corrected  immediately,  to  major  ommissions  (OPR  DISC  and  KEY  ON  errors) 
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which  produce  extensive  delays  and  cause  subscribers  to  miss  communications. 

On  the  other  hand,  operator  maintenance  can  be  accomplished  in  a relatively 
short  time  with  few  errors. 

Operators  detected  most  of  the  errors  observed  during  CCD  operations 
(link  establishment  and  key  actuation  errors)  and  corrected  them  after  some 
delay.  In  some  cases,  such  as  activation  of  an  incorrect  key,  the  CCU 
operator  rapidly  discovered  the  error  and  corrected  it  with  little  disruption 
of  his  task.  Other  errors,  such  as  verifying  connection  with  each  link 
subscriber  individually,  rather  than  verifying  all  subscribers  simultaneously, 
delayed  the  task  excessively  but  did  not  prevent  its  completion. 

However,  several  error-producing  problems  are  expected  to  decrease  the 
reliability  and  availability  of  communications  through  the  CCU.  In  particular, 
the  operator  disconnect  (OPR  DISC)  error,  which  occurred  33%  of  the  time, 
can  be  expected  to  reduce  the  reliability  of  communications  if  a subscriber 
must  speak  to  (ring  in)  the  CCU  operator.  In  this  case,  the  CCU  operator 
must  remember  to  disconnect  the  subscriber  from  the  Operator  Circuit.  There 
is  no  feedback  to  tell  the  CCU  operator  if  he  has  failed  to  push  the  OPR  DISC 
keys.  Long  delays  can  be  expected  in  restoring  a subscriber  to  an  intended 
circuit.  This  error  will  have  a noticeable  impact,  not  only  in  terms  of  its 
consequence  but  also  in  terms  of  its  frequency  of  occurrence.  After  all  CCU 
subscribers  have  been  initially  connected  and  the  subscribers  begin  to  request 
a variety  of  changes  in  the  configurations  of  communication  nets  and  links 
from  the  CCU  operator,  he  must  push  OPR  DISC  frequently.  This  high  task 
performance  rate,  plus  the  high  rate  of  errors  (33%)  when  the  task  is  per- 
formed, will  create  many  opportunities  for  the  error  to  occur.  These  errors 
will  reduce  system  reliability,  but  they  will  also  make  the  operator  more 
aware  of  their  likelihood,  thereby  offsetting  some  of  the  negative  impact  on 
rel iability. 

The  KEY  ON  error  is  not  expected  to  occur  frequently;  thus,  an  operator 
will  not  learn  as  rapidly  to  guard  against  its  occurrence  so  quickly.  As  a 
result,  the  KEY  ON  error  can  be  expected  to  have  a noticeable  impact  on 
system  availability  since  there  may  be  long  delays  in  establishing  communi- 
cation among  CCU's  in  several  TACFIRE  units. 


Impact  Upon  System  Effectiveness 

Since  the  CCU  operator's  human  performance  time  goals  have  not  been 
specified,  the  impact  of  the  operator's  performance  times  and  error 
frequencies  on  overall  system  effectiveness  must  be  estimated  from  his  test 
performance  with  identified  missions. 

The  performance  requirements  for  TACFIRE 's  automated  data  systems  are 
described  in  terms  of  mission  response  times  under  assumed  conditions.  One 
such  mission  would  be  to  issue  a firing  order  to  a particular  artillery  bat- 
tery in  response  to  a firing  request  from  a forward  observer.  This  mission  can 
be  assumed  to  occur  within  10  seconds.  In  this  case,  the  mission  response 
time  is  measured  from  the  time  the  forward  observer  pushes  the  "TRANSMIT" 
button  on  his  message-entry  device  until  the  firing  order  is  displayed  on 


133 


the  battery  message  printer.  The  mission  time  includes  data  transmission, 
computer  calculations,  active  acceptance  of  the  computed  firing  order  by  the 
Fire  Direction  Officer  (FDO),  and  the  specified  probability  of  the  system 
being  down  for  maintenance.  The  mission  conditions  also  assume  that  all 
required  communication  circuits  have  been  pre-established.  The  specified 
mission  response  time  does  not  include  the  time  required  for  the  forward 
observer  to  input  the  target  data  into  his  message-entry  device,  any  time 
required  by  the  ACC  operator  to  select  alternative  firing  orders,  or  any 
time  required  by  the  battery  personnel  to  implement  the  firing  order.  These 
latter  performance  times  can  be  expected  to  be  several  times  longer  than  the 
mission  time  itself,  since  the  previously  identified  baseline  task  (setting 
up  a net)  requires  a mean  performance  time  of  33.6  seconds. 

Given  this  baseline  task  as  a representative  example  of  the  time 
required  for  the  CCD  operator  to  establish  a communication  circuit,  it  can 
be  seen  that  the  sample  mission  response  time  would  be  increased  340%,  from 
10  seconds  to  43.6  seconds,  if  the  mission  had  been  defined  to  include  any 
time  required  to  establish  communications.  Obviously,  some  communications 
circuits  would  have  been  set  up  by  the  time  any  artillery  firing  order 
would  be  required.  Therefore,  a more  complete  assessment  of  mission  response 
time  would  include  some  estimate  of  the  probability  that  the  required 
communication  circuit  is  not  yet  established.  For  the  purpose  of 
illustration,  assume  this  probability  to  be  10%.  Thus  the  estimated  response 
time  would  be  calculated  as: 

Response  time  = Data  transmission  and  calculation  time 
+ communication  set-up  time 
= 10  sec.  + (.10)  (33.6  sec.) 

= 10  sec.  + 3.4  sec. 

= 13.4  seconds 

Thus,  if  nets  are  assumed  to  be  established  during  90%  of  the  missions,  a 
34%  increase  in  mission  response  time  is  found,  compared  to  conditions  of 
100%  pre-established  nets. 

If  the  mission  could  not  be  accomplished  over  the  standard  nets  and  a 
link  had  to  be  established,  the  mission  response  time  would  become 
considerably  longer.  The  average  time  required  by  the  test  subjects  to  set 
up  a link  was  112  seconds.  This  task  includes  not  only  connecting  the 
subscribers,  but  also  requires  answering  a ring-in  from  a subscriber 
requesting  that  the  link  be  established.  The  task  of  establishing  a link 
is  also  likely  to  include  the  OPR  DISC  error  (33%  of  the  time)  where  the 
operator  fails  to  push  the  OPR  DISC  buttons  after  answering  the  ring  thus 
keeping  the  subscriber  on  the  operator  circuit.  Thus  the  link  set-up  time 
can  be  analyzed  as  a combination  of  times  as  follows: 

Link  set  up  = connect  circuit 
(time) 

+ answer  ring 

+ (correct  OPR  DISC  error)  x (prob.  of  error) 

The  task  of  establishing  a link  requires  230%  more  time-112  seconds  as 
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compared  to  34  seconds-than  the  highly  similar  task  of  establishing  a net. 
Thus,  if  the  firing  order  must  be  sent  over  a link  which  has  not  yet  been 
established,  the  mission  response  time  would  increase  from  the  original  10 
seconds  to  122  seconds. 

When  a TACFIRE  mission  includes  communication  between  two  fire  direction 
centers,  a single  net  can  be  established,  with  the  CCU  from  the  second  fire 
direction  center  considered  as  a subscriber  to  the  first  CCU.  For  example, 
such  a configuration  would  occur  if  one  battalion  TACFIRE  computer  was 
inoperative  and  a second  battalion  TACFIRE  computer  operated  as  the  Lateral 
Back-Up  (LBU).  The  mission  response  time  for  receiving  the  firing  request 
from  the  forward  observer  and  issuing  a firing  order  can  be  assumed  to  be 
14  seconds,  if  all  communication  nets  are  pre-established.  As  illustrated 
earlier,  if  the  communication  nets  are  assumed  to  be  established  90%  of 
the  time,  the  response  time  for  the  present  mission  increases  from  14  seconds 
to  17.4  seconds.  However,  in  the  present  case,  there  is  a high  probability 
that  the  CCU  operator  will  fail  to  press  the  KEY  ON  buttons,  failing  to 
establish  communication  between  the  two  CCU's.  The  test  participants 
failed  to  press  the  KEY  ON  buttons  in  80%  of  the  required  occasions.  The 
time  required  by  the  operator  to  identify  and  correct  the  KEY  ON  error  will 
vary  considerably,  depending  primarily  on  the  operator's  training  and 
experience.  For  the  purpose  of  the  HFE  test,  timing  was  continued  until 
the  error  was  corrected  or  until  180  seconds  had  elapsed.  Given  this 
restriction,  the  average  time  to  verify  a connection  with  another  CCU 
was  102  seconds,  as  compared  with  29  seconds  to  verify  connections  with 
all  other  types  of  subscribers.  Given  these  data,  the  time  required  to 
complete  the  mission  of  generating  a firing  order  and  transmitting  it 
between  two  TACFIRE  computer  centers  may  be  calculated  as  follows: 

Mission  response  time 

= T^  + P(C)  [(Tj.)  + P (E)  (T^)] 

= 14  seconds  + (.1)  [(33.6  sec.)  + (.8)  (102  sec.)] 

= 14  seconds  +11.5  seconds 
=25.5  seconds 

Where 

Jj  = Data  Transmission  and  Calculation  time 
T^  = Communication  set-up  time 
T^  = Error-correction  time 

P(C)  = Probability  communications  will  NOT  be  established 
P(E)  = Probability  of  KEY  ON  error 

Assuming  that  communication  nets  would  be  pre-established  90%  of  the 
time,  and  given  the  restricted  time  allowed  for  detecting  the  KEY  ON  error 
in  the  HFE  test,  the  mission  response  time  is  increased  by  82%,  from  14 
seconds  to  25.5  seconds.  If  the  error-correction  time  had  not  been 
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arbitrarily  restricted  to  180  seconds  during  testing,  a significantly 
larger  increase  in  mission  response  time  would  be  found. 

This  HFE  test  has  demonstrated  that  the  time  required  by  the  CCU 
operator  to  perform  his  normal  duties  contributes  significantly  to  the 
mission  response  times  defined  for  the  TACFIRE  system.  However,  it  must 
be  stressed  that  the  TACFIRE  mission  response  time  performance  requirements 
were  expressly  defined  to  exclude  human  performance,  including  message  entry, 
selection  of  alternative  computer-generated  firing  directives,  responding 
to  computer-printed  orders,  etc.  Given  this  limited  definition  of  required 
response  times,  the  task  of  establishing  communication  nets  was  shown  to 
provide  a baseline  for  comparing  other  CCU  operator  tasks.  This  test  has 
demonstrated  that  operator  errors,  such  as  failing  to  push  the  correct 
buttons,  can  increase  a task  performance  time  as  much  as  80%  and  that  a 
poorly-understood  task,  such  as  connecting  a link,  can  increase  the  task 
time  by  230%  as  compared  with  the  similar  baseline  task. 


Solutions  for  Improving  Human  Performance 

Design  changes  have  been  recommended  for  the  error-producing  problems 
noted  in  the  task,  since  a one-time  investment  in  redesign  should  eliminate 
the  errors,  whereas  recurring  training  costs  would  be  expected  to  reduce  but 
not  necessarily  eliminate  the  errors.  In  particular,  software  changes  are 
recommended  to  eliminate  the  OPR  DISC  and  KEY  ON  errors.  Key  redesign,  such 
as  adding  a sounding  relay,  is  recommended  to  eliminate  key  actuation  errors. 
Wider  key  spacing  is  recommended  to  eliminate  simultaneous  key  pressing. 

Increased  training  is  recommended  to  reduce  the  long  time  delays  noted 
in  the  CCU  operations.  These  latter  tasks  included  errors  which  could  be 
eliminated  by  the  recommended  design  changes.  But  the  task  performances 
also  included  enough  hesitations  to  suggest  that  thorough  training,  partic- 
ularly, "hands-on"  training  with  task  scenarios,  will  dramatically  improve 
performance  times.  The  observed  time  delays  will  also  be  reduced  with 
increasing  on-the-job  training  as  a new  CCU  operator  becomes  familiar  with 
TACFIRE  team  procedures. 


CONCLUSIONS 


Test  Findings 

The  major  HFE  test  findings  are  summarized  below: 
a.  Problems: 

1.  Depressing  the  keys  provided  the  operator  with  inadequate 
sensory  feedback  to  determine  whether  the  key  had,  in  fact, 
been  actuated. 
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2.  The  center-to-center  spacing  of  the  keys  was  closer  than 
MIL-STD-1472B  (36),  specifies. 

3.  The  operator  did  not  have  sufficient  space  or  a work 
surface  to  record  required  information. 

4.  Headset  jacks  and  connections  were  not  labeled  causing 
operators  to  connect  the  headsets  to  an  inappropriate  jack. 

5.  The  keyboard  is  designed  for  initial  establishment  of  nets; 
it  is  not  appropriately  designed  for  performing  CCU  tasks 
during  operations. 

b.  Performance  errors  occurred  during  the  following  tasks: 

1.  Disconnecting  subscribers  from  the  CCU  operator  circuit 
(Operator  disconnect  error). 

2.  Establishing  nets. 

3.  Setting  up  links. 

4.  Disconnecting  links. 

5.  Attempting  to  establish  communication  with  another  CCU 
(KEY  ON  error). 

c.  Equipment  and  task  group  incompatibilities: 

1.  The  CCU  operator  can  not  get  to  or  leave  his  work  station 
without  interfering  with  the  ACC  operator. 


Implications  for  System  Performance 

The  results  of  this  HFE  test  demonstrate  that  a trained  CCU  operator 
can  rapidly  and  accurately  attach  the  communication  wires  to  the  CTB,  estab- 
lish the  initial  net  and  link  connections  among  subscribers,  and  perform  the 
maintenance  checks  on  the  CCU  circuits.  However,  the  test  demonstrated  that 
long  time  delays  and  many  errors  can  be  expected  to  occur  when  the  CCU  is 
used  as  a communications  switchboard  and  the  CCU  operator  is  required  to 
reconfigure  nets  and  links  in  response  to  subscriber  requests.  During  these 
latter  CCU  operations,  the  operator's  errors  and  time  delays  will  reduce  the 
reliability  and  availability  of  TACFIRE  communications,  particularly  when  a 
CCU  operator  first  joins  a TACFIRE  team.  The  recommended  design  changes  are 
expected  to  reduce  the  operator's  errors  significantly,  and  the  recommended 
training  improvements  will  improve  his  performance  times. 


Recommended  Changes 

The  following  changes  are  recormtended  to  eliminate  sources  of  error  and 
to  improve  overall  system  reliability  and  effectiveness: 
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1.  Allocate  to  software  the  tasks  of  (a)  disconnecting  subscribers 
from  the  operator's  circuit,  and  (b)  activating  the  "KEY  ON" 
function  before  communicating  with  another  CCD. 

2.  Expand  the  training  program  to  include  hands-on  practice  in 
performing  net  and  link  tasks.  Use  scenarios  similar  to  those 
used  in  testing  to  insure  that  trainees  can  perform  all  required 
tasks. 

3.  Redesign  the  keys  so  that  they  provide  definite  actuation  feed- 
back. 

4.  Label  the  communication  headset  jacks  to  reduce  the  probability 
of  the  operator  connecting  the  headset  to  the  wrong  jack,  or 
eliminate  the  unnecessary  jack. 

5.  Provide  a working/writing  table  for  the  operator. 

6.  Increase  key  spacing  to  conform  to  MIL-STD-1472B  (36). 

7.  To  improve  the  CCU  operation  during  missions,  the  keyboard  should 
be  redesigned  on  the  basis  of  expected  frequency  and  sequence  of 
use  data. 
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IMPLICATIONS  OF  HUMAN  PERFORMANCE  TESTS 


The  introduction  to  this  report  explained  why  it  is  necessary  to  have  HFE 
testing  in  materiel  development  programs  and  showed  how  DI-H-1334A  systemati- 
cally and  effectively  incorporates  HFE  data  into  the  Army's  materiel  acquisi- 
tion cycle.  Succeeding  chapters  gave  details  of  how  to  conduct  and  report  HFE 
tests,  with  two  complete  sample  test  reports  showing  how  tests  should  be 
conducted  to  meet  DI-H-1334A  requirements.  However,  the  HFE  test  report  is 
just  one  part  of  the  entire  system  development  program,  and  it  must  be  inte- 
grated into  that  program  effectively.  The  paragraphs  in  this  section  consider 
how  HFE  test  data  should  be  used,  when  HFE  tests  should  be  conducted,  and 
what  problems  should  be  expected  in  conducting  these  tests. 


HFE  Data  in  the  Materiel  Development  Program 

System  reliability  is  often  defined  narrowly  (e.g.,  59,  p.  1)  as  a 
combination  of  the  reliability  of  the  hardware  components.  This  definition 
of  reliability  follows  naturally  when  a system  is  defined  for  a contractor 
in  terms  of  the  deliverable  items  of  equipment.  However,  the  user  defines 
the  system  to  include  not  only  the  equipment,  but  also  the  personnel  who 
operate  and  maintain  the  equipment  and  their  training.  Thus  the  user  is 
concerned  with  the  field  effectiveness  of  the  man-machine  system  which 
includes  not  only  the  reliability,  maintainability,  and  availability  of 
the  equipment  but  also  the  capabilities  of  the  personnel  to  operate  and 
maintain  the  equipment  within  prescribed  time  limits  and  acceptable  error 
rates.  Clearly,  any  attempt  at  effectiveness  assessment  which  does  not 
include  the  human  element  is  likely  to  be  incomplete  and  to  produce  an 
inflated  estimate  of  the  effectiveness  of  a system. 

The  human  factors  test  is  that  step  of  the  development  program  which 
measures  the  operator's  performance.  It  has  two  purposes:  (1)  to  assess 
what  the  human  operator's  performance  contributes  to  overall  system 
performance,  and  (2)  to  identify  the  causes  of  human  error  or  substandard 
performance.  Thus,  the  test  data  are  used  directly  in  calculating  indices 
of  reliability  (failure  rate,  mean  time  between  failures,  time  to  repair, 
etc.),  as  well  as  indices  of  effectiveness  (accuracy,  throughput  rates, 
etc.).  Indirectly,  HFE  data  provide  a means  of  identifying  and  correcting 
the  causes  of  human  performance  problems  and  errors.  The  HFE  test  report 
analyzes  hunfian  performance  empirically  in  terms  of  performance  times  and 
error  rates,  and  it  gives  qualitative  descriptions  of  the  factors  that 
contribute  to  errors  and  slow  performance  times. 

But  merely  describing  the  factors  that  produce  errors  --  such  as 
insufficient  training  for  Task  X,  inappropriately  allocating  Function  Y 
to  the  human  operator,  or  improperly  placing  Display  Z --  is  not  enough. 
Identifying  an  error  is  only  the  first  step  in  eliminating  it  or  reducing 
its  frequency.  If  the  HFE  test  report  is  to  make  an  effective  contribution 
to  the  system  development  program,  it  must  also  suggest  ways  the  observed 
problems  or  errors  can  be  solved  or  corrected.  To  be  most  effective,  the 
report  should  provide  several  alternative  solutions  and  estimate  how 
effectively  they  can  alleviate  the  problem.  Engineering  and  cost  analysis 
personnel  can  then  evaluate  the  alternative  solutions  and  give  the  program 
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management  the  information  needed  to  make  a decision. 

The  stage  of  system  development  also  affects  how  HFE  data  will  be  used. 
In  the  experimental  prototype  phase,  testing  focuses  on  establishing  whether 
human  performance  is  feasible.  Data  from  early  testing  --  frequency  and 
sequence  of  use,  and  criticality  of  equipment  components  --  is  used  to 
evaluate  how  appropriately  functions  have  been  allocated,  and  to  lay  out 
the  workspace.  Time  and  error  data  from  early  testing  can  be  used  for 
preliminary  estimates  of  human  reliability  and  system  effectiveness.  Figure 
21  shows  that,  as  system  development  progresses,  more  and  more  parameters 
are  measured  empirically  and  used  to  derive  reliability  and  effectiveness 
estimates.  In  addition,  as  shown  in  Figure  22,  progressively  more  confidence 
can  be  placed  in  these  estimates  as  development  proceeds.  The  more  input 
parameters  that  are  measured,  the  more  accurately  system  effectiveness  can 
be  predicted. 

In  the  later  stages  of  system  development,  testing  focuses  on  assessing 
the  operator's  ability  to  perform  his  assigned  tasks.  Time  and  error  data, 
and  the  factors  that  predisposed  the  operator  to  err,  are  used  in  estimating 
human  reliability  and  trading  off  alternative  ways  of  eliminating  factors 
that  predispose  the  operator  to  err.  It  is  necessary  to  quantify  error  rates  and  list  the  alternative 
solutions  before  their  costs  can  be  traded  off  against  the  improved  system  effectiveness  they  offer. 


Scheduling  Human  Performance  Testing  During  System  Development 

Figure  23  shows  the  potential  impact  that  human  factors  design  and 
testing  can  have  at  different  stages  of  system  development.  Not  surpris- 
ingly, it  costs  progressively  more  to  make  HFE  changes  in  later  stages  of 
development.  Yet  changes  recommended  during  later  development  are  also 
more  likely  to  improve  system  effectiveness,  since  they  are  based  on  more 
precise  data.  Unfortunately,  the  probability  that  the  program  manager 
will  adopt  recommended  HFE  changes  decreases  as  the  design  becomes  defined 
more  rigidly. 

Figure  23  also  shows  that  the  most  cost-effective  time  to  detect  and 
correct  human  errors  or  substandard  performance  times  is  early  in  the 
development  cycle.  Army  leaders  understand  this  relationship  (54,55),  and 
it  can  be  expected  that  cost  effective  HFE  tradeoffs  will  become  a more 
prominent  feature  of  system  development  in  the  future.  Human  factors 
testing  produces  the  most  cost-effective  problem  solutions  when  it  is  done 
in  the  experimental -prototype  stage,  because  it  is  economical  to  redesign 
equipment  before  hardware  has  been  constructed.  These  design  modifications 
are  one-time  costs;  whereas,  the  cost  of  upgrading  selection  and  training 
continues  throughout  the  life  of  the  system. 

Although  human  factors  testing  must  be  conducted  early  in  system 
development  to  produce  timely  recommendations  for  economic  rectification 
of  human  performance  problems,  HFE  testing  will  also  be  required  in  later 
system  testing.  HFE  tests  in  "advanced  development"  and  "engineering 
development"  are  conducted  with  personnel  and  equipment  which  are  more 
representative  of  operational  conditions.  HFE  tests  in  later  developmental 
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PERCENTAGE  OF  PARAMETERS 
EMPIRICALLY  MEASURED 
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EXPERIMENTAL  ' ADVANCED  ' ENGINEERING 
PROTOTYPE  DEVELOPMENT  DEVELOPMENT 

STAGE  OF  DEVELOPMENT 


Figure  21.  Performance  parameters  measured  during  system  development 
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EXPERIMENTAL  ADVANCED  ENGINEERING 

PROTOTYPE  DEVELOPMENT  DEVELOPMENT 

STAGE  OF  DEVELOPMENT 


Figure  22.  Confidence  limits  of  performance 
measured  during  system  development 
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stages  are  also  necessary  to  determine  the  adequacy  of  personnel  selection 
and  training.  Earlier  HFE  tests  may  be  able  to  provide  little  data  with 
regard  to  the  appropriateness  or  completeness  of  selection  or  training, 
particularly  for  an  entirely  new  system,  since  training  and  selection 
programs  may  not  be  completely  defined  prior  to  DT  II.  Therefore,  later 
HFE  tests  will  highlight  potential  deficiencies  which  may  not  have  been 
observable  in  the  earlier  tests.  As  implied  by  Figure  20,  the  effects  of 
expected  use  conditions,  such  as  environmental  conditions,  can  be  most 
accurately  assessed  in  HFE  tests  in  later  stages  of  development.  System 
effectiveness  will  also  be  estimated  more  accurately  from  later  tests,  since 
it  is  then  based  on  the  performance  data  from  operators  who  resemble  the 
eventual  users  more  closely. 

Regarding  the  scheduling  of  HFE  tests  during  system  development.  Army 
test  and  evaluation  procedures  (76)  require  that  human  factors  test  data 
be  included  at  all  In-Process  Reviews  (IPR)  or  Army  System  Acquisition 
Review  Councils  (ASARC).  Use  of  these  data  permit  an  assessment  of  human 
performance  to  be  considered,  along  with  other  design  data,  so  that 
appropriate  decisions  can  be  made.  System  development  programs  should 
therefore  include  human  factors  testing  as  an  integral  portion  of  develop- 
mental testing.  As  specified  in  AR  70-10,  (76)  the  HF  testing  can  be 
conducted  by  the  contractor  as  part  of  DT  I and  II  to  provide  an  economical 
testing  program.  The  human  performance  data  collected  and  reported  according 
to  the  requirements  of  DI-H-1334A  can  be  used  to  provide  the  human  factors 
data  required  by  the  system  evaluator.  Depending  upon  the  nature  of  the 
program,  a single  DI-H-1334A  may  be  submitted  or  a series  of  tests  may  be 
prepared  starting  soon  after  contract  award  and  continuing  throughout  the 
coordinated  test  program. 

Applying  DI-H-1334A  throughout  development,  from  the  system's  inception 
through  operational  deployment,  is  a systematic  way  to  integrate  human 
factors  considerations  into  materiel  development  programs. 


Solving  the  Problems  of  HFE  Testing 

In  conducting  the  two  sample  HFE  tests,  most  of  the  problems  arose 
from  administration,  planning,  and  assessing  the  participants'  training. 
These  problems  are  not  peculiar  to  the  two  specific  examples;  they  can  be 
expected  to  occur  unless  precautions  are  taken  to  prevent  them.  Hence, 
the  problems  the  authors  encountered,  and  the  solutions  that  were  developed, 
will  be  described  briefly  here. 

The  main  administrative  problems  in  implementing  DI-H-1334A  were 
coordinating  all  of  the  management  and  technical  personnel,  and  scheduling 
the  test  milestones.  These  problems  are  like  those  of  coordinating  any 
test  for  a multidisciplinary  materiel  development  program.  They  included 
activities  such  as  gathering  descriptions  and  drawings  of  equipment  from 
engineering  departments,  obtaining  definitions  of  system  goals  and  descrip- 
tions of  system  operation  from  system  analysis  personnel,  and  scheduling 
the  construction  or  acquisition  of  test  equipment. 
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A good  solution  for  the  problems  of  administration  is  to  appoint  a 
single  person  as  test  manager  with  overall  responsibility  for  the  HFE 
test  (even  if  this  person  also  has  other  responsibilities  in  the  materiel 
development  and  testing  program).  The  centralization  of  responsibility  for  human 
factors  testing  in  one  person  will  facilitate  planning  and  coordination. 

The  test  manager  should  have  the  knowledge  and  experience  to  carry  out  the 
tasks  of  a manager  of  a small  project  (1).  The  test  manager  should  also 
have  sufficient  technical  knowledge  and  experience  in  human  factors  testing 
to  be  able  to  perform  or  direct  all  of  the  activities  shown  previously  in 
Table  1. 

Planning  the  HFE  test  is  a major  activity.  Thus,  the  test  manager 
contributes  significantly  to  the  success  of  the  test  by  the  manner  in  which 
he  plans  the  test  activities.  It  is  important  to  schedule  DI-H-1334A  test 
activities  carefully  to  avoid  major  problems.  The  checklist  previously 
illustrated  in  Table  1 provides  a systematic  sequence  of  activities  that 
must  be  scheduled.  The  following  items  must  be  completed  prior  to  testing 
to  insure  that  all  scheduled  activities  occur  in  a timely  manner: 

1.  Mockups  must  be  prepared  and  checked  out  sufficiently  in 
advance  to  permit  whatever  layout  changes  are  found  to  be 
necessary. 

2.  All  necessary  test  equipment  and  instrumentation  must  be 
obtained  and  calibrated. 

3.  Test  scenarios  need  to  be  developed  and  verified. 

4.  Preplanning  at  the  test  location  is  essential,  including  test 
dry  runs  to  assure  that: 

a.  all  instruments  function  correctly, 

b.  test  personnel  understand  what  they  are  supposed  to  do  in 
obtaining  and  recording  test  data, 

c.  test  personnel  can  see  what  is  happening  and  record  data 
without  interfering  with  test  participants. 

Selecting  and  screening  the  participants  can  be  troublesome,  particu- 
larly in  early-development  phases  when  the  training  and  selection  criteria 
have  not  been  defined  completely.  Nevertheless,  it  is  essential  to  use 
adequately  trained  participants,  since  poorly  trained  operators  can  make 
the  system  seem  much  less  effective  than  it  actually  is. 

The  intent  of  DI-H-1334A  is  to  use  test  participants  who  resemble,  as 
closely  as  reasonably  possible,  the  eventual  system  personnel.  Thus,  efforts 
must  be  made  to  measure  the  capabilities  of  the  test  participants  with 
respect  to  those  characteristics  deemed  necessary  for  the  eventual  personnel 
(12).  In  addition  to  basic  capabilities,  such  as  intelligence  and  visual 
acuity,  the  test  participants'  training  must  be  assessed.  The  measurement 
of  this  training  appears  to  be  one  of  the  most  difficult  problems  in 
conducting  a DI-H-1334A  test  and  can  best  be  solved  by  preparing  testing 
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materials  or  short  job  tests  which  adequately  measure  thp  participants' 
abilities  to  perform  the  essential  tasks  identified  in  the  task  analysis. 

The  results  of  these  tests  will  determine  the  proficiency  of  the  test 
participants  and  indicate  any  need  for  refresher  training  before  the  HFE 
testing  begins.  If  the  training  tests  or  expert  opinion  indicated  the  test 
participants  are  not  sufficiently  prepared  to  operate  or  maintain  the 
system  equipment  properly,  extra  time  must  be  allocated  to  provide  sufficient 
refresher  training. 


Human  Factors  in  System  Development 

Normally,  the  person  operating  the  system  doesn't  want  to  make  mistakes, 
and  it  is  the  design  of  the  equipment  which  sets  his  basic  error  frequency 
(41).  It  has  been  reported  that  50  to  70  percent  of  all  failures  in  major 
weapons  and  space  systems  are  human  initiated,  placing  "human  error  ahead 
of  design  error,  component  unreliability,  and  lapses  in  quality  control  in 
manufacturing..."  (15).  Clearly,  it  is  important  for  successful  system 
operation  to  identify  the  sources  of  human  error  and  to  provide  solutions 
for  eliminating  the  errors  as  soon  as  possible  in  the  system  development 
program. 

Human  factors  personnel  can  provide  timely  analyses  and  design  recom- 
mendations with  regard  to  the  required  human  performance,  the  personnel 
selection  and  training  procedures,  and  the  man-machine  interface  design. 
However,  human  factors  tests  must  be  conducted  to  answer  the  questions  "How 
well  does  the  equipment  perform  in  the  hands  of  the  soldier  in  the  field?" 
Data  item  DI-H-1334A  provides  a systematic  methodology  for  including  human 
factors  data  throughout  a system  development  cycle.  Properly  used  at  the 
appropriate  times,  this  data  item  provides  a cost-effective  method  for 
evaluating  and  improving  human  performance  contributions  to  overall  system 
effectiveness. 
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APPENDIX  1 


ecu  OPERATIONS  TRAINING  TEST 


Subject  

Date  ! / 76  Time 


ecu  OPERATORS  TEST 

How  many  subscribers  are  possible  with  the  CCU? 


net? 


DDT 

Wire 

Radio 

Local 

4 

8 

8 

10 

8 

10 

16 

16 

10 

32 

20 

20 

16 

40 

32 

32 

is  the 

maximum  number  of 

subscribers 

that  can  be  inc 

DDT 

Wire 

Radio 

Local 

1 

1 

1 

2 

2 

4 

3 

8 

4 

6 

6 

10 

6 

10 

8 

20 

3.  Can  an  AM  radio  and  an  FM  radio  be  connected  to  the  same  net? 

True  False 

4.  How  many  nets  can  be  set  up  on  the  CCU?  (Don't  count  the  net  which 
holds  the  unassigned  subscribers.) 

5.  On  what  net  are  unassigned  subscribers  parked? 

6.  The  intercom  (COM)  is  used  primarily  to  communicate  with 


DDT 

Wire 

Local 

Radio 

RCMU 


7.  The  operator  circuit  is  used  to  answer  calls  from  all  of  the  following 
subscribers  except  one.  Identify  the  one  exception. 

Wire 

Local 

Radio 

RCMU 
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8.  List  all  five  types  of  circuits  available  on  the  CCU. 

1. 

2. 

3. 

4. 

5. 

9.  Digital  data  traffic  can  be  transmitted  through  the  CCU  only  in  the 
AUTO  mode. 

True  False 

10.  List  the  sequence  of  buttons  pressed  to  put  the  following  subscribers 
on  link  3.  (Put  the  subscribers  on  the  link  in  the  following  order:) 

1 . Wi  re  1 

2.  Wire  5 

3.  Radio  2 

4.  Local  5 

5.  Local  8 

11.  The  following  displays  are  flashing: 

RING  2 

RAD 

RING  2 3 

LOCAL  STATUS/RING  19 

List  the  sequence  of  buttons  pressed  to  answer  the  ring  from  Radio  3. 


Now  list  the  sequence  of  buttons  pressed  to  answer  Local  19. 

(Remember  you  had  responded  to  Radio  3 just  before  answering  Local  19.) 
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12.  WIRE  7,  RADIO  2,  and  LOCAL  8 are  connected  on  Link  1.  What  happens 
if  WIRE  7 rings  in  on  the  link? 

1.  Wire  7 is  disconnected  from  Link  1. 

2.  RING  display  shows  7 

3.  All  subscribers  on  Link  1 are  disconnected. 

4.  None  of  the  above. 

13.  On  what  circuit  is  a subscriber  located  after  the  CCU  operator  presses 
ANS? 

14.  On  what  circuit  is  a subscriber  located  after  the  CCU  operator  presses 
OPR  and  DISC? 

15.  Power  to  the  RCMU  is: 

a.  Provided  by  a 28  volt  nickel  cadminum  battery. 

b.  Provided  by  the  CCU. 

c.  RCMU  is  a passive  monitoring  device;  no  power  is  provided  to  it. 

d.  28  volts  AC  at  J1 . 

16.  When  power  is  turned  on  at  the  CCU,  blinking  error  code  88  indicates 
that: 

a.  Ring  in  and  data  control  processing  is  ready  to  be  processed. 

b.  Self-test  passed  successfully. 

c.  Net  files  are  filled. 

d.  Link  files  are  filled. 

17.  If  prime  power  coming  from  the  PCG  to  the  CCU  were  lost: 

a.  The  CCU  would  remain  fully  operational  for  at  least  ten  minutes. 

b.  The  CCU  memory  would  retain  the  Net  and  Link  set-ups  for  ten 
minutes. 

c.  Error  code  84  will  blink  continuously  for  ten  minutes. 

d.  The  RCMU  will  continue  to  function  because  it  is  a passive 
device. 


156 


18.  The  maximum  number  of  Radio  subscribers  per  net  are: 

a.  6 Radios,  AM  and/or  FM. 

b.  3 Radios,  AM  only. 

c.  3 Radios,  FM  only. 

d.  3 Radios,  AM  and/or  FM. 

19.  Which  of  the  following  is  not  a communication  capability  of  the  CCU? 

a.  Manual  cordless  switchboard. 

b.  Local  Intercom. 

c.  Integrated  Multi-Media  nets. 

d.  Integration  Net-Link  facility. 

20.  When  establishing  wire  communication  between  CCU's: 

a.  KEY  ON  must  be  used  at  both  CCU's. 

b.  Only  local  subscriber  circuits  are  used. 

c.  KEY  ON  is  required  if  trouble  is  experienced  with  polarity. 

d.  Local  subscriber  circuits  may  be  used  if  they  are  BOX  circuits. 

21.  The  major  units  of  the  CCU  Central  Logic  Unit  are: 

a.  lOU,  CPU,  ROM,  RAM. 

b.  Timing  Unit,  Central  Processing  Unit,  CCU  Analog  Card  Interface. 

c.  Timing  Unit,  CPU,  lOU. 

d.  Timing  Unit,  lOU. 

22.  To  move  Net  2 Local  Subscriber  10  to  Net  3 the  most  logical  keyboard 
sequence  would  be: 

a.  NET,  2,  LCL,  10,  DISC,  NET,  3,  CONN. 

b.  NET,  2,  LCL,  1,  0,  DISC,  NET,  3,  LCL,  1,  0,  CONN. 

c.  NET,  3,  LCL  1,  0,  CONN. 

d.  CONN,  NET  3,  LCL,  10. 


i 

I 
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23.  To  disconnect  NET  2 from  the  CCU  the  operator  would  depress  the 
following  switches: 

a.  NET,  2,  OP,  DISC,  DISC,  DISC 

b.  NET,  2,  DISC,  DISC,  DISC 

c.  NET,  2,  WIRE  2,  DISC,  RAD  2,  DISC  LOCAL  2,  DISC 

d.  NET,  DDT  2,  DISC,  RAD  2,  DISC,  LOCAL  2,  DISC  WIRE  2 DISC 

24.  To  connect  a Radio  subscriber  to  a net  the  operator  would  take  the 

following  key  sequence: 

a.  RADIO  NO.  NET.  NO. 

b.  NET,  NO, .RADIO  NO. 

c.  RADIO,  NO,  NET,  NO.  CONN. 

d.  NET,  NO,  RADIO,  NO.  CONN. 

25.  Whenever  the  NET  Window  in  the  display  is  not  blank  or  zero  the 
following  switches  may  be  depressed: 

a.  MON,  TALK,  OFF. 

b.  MON,  TALK,  OFF,  XMIT. 

c.  MON,  TALK,  OFF,  XMIT,  MAN  and  AUTO. 

d.  MON,  TALK,  OFF,  MAN  and  AUTO. 

26.  The  RCMU  can  monitor: 

a.  8 nets. 

b.  7 nets  and  one  local  line. 

c.  6 nets,  a local  line,  and  an  intercom  line. 

d.  Only  one  net  at  a time. 
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27.  The  KEY  ON  feature  is  indicated  by  a decimal  point  to  the  right  of 
the  respective: 

a.  Local  subscriber. 

b.  FM  radio  subscriber. 

c.  AM  radio  subscriber. 

d.  Wire  subscriber. 

28.  The  intercom  circuit  is: 

a.  Used  between  the  CCU  operator  and  local  subscribers. 

b.  A 5-party  line  with  4 RCMU's  and  a CCU. 

c.  Used  in  conjunction  with  the  Operator  circuit  to  make  calls  to 
the  FO's. 

d.  Essentially  a one-way  circuit. 

29.  Power  up  on  the  CCU  initiates: 

a.  A cold  start  test  of  the  CCU  and  RCMU. 

b.  An  initialization  self-test  and  then  a cold  start  self-test  of 
the  CCU. 

c.  An  initialization  self-test  of  the  CCU  and  RCMU. 

d.  A cold  start  test  of  the  multi-media  nets  only. 

30.  The  maximum  number  of  local  telephone  subscribers  on  any  one  net  are: 

a.  8 VOX 

b.  20 

c.  12 

d.  6 on  two  wire 
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APPENDIX  2 

SAMPLE  BEHAVORIAL  CHECKLIST 
(CCD  Test  Segment  4) 
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test  SEGtCKT  4 (CONTINUED) 


SUBJECT  NAH£_ 
DATE 


24.  BN  CDR  calls  in  Net  1 
DDT  DONN,  exchange  wit 
F3  Net  6 


END  TIME 


INFORMATION 


25.  Verify  Nets 
Net  1 
Net  6 


26.  Call  in  from  MASH 
DISC,  from  Net  6 


27.  Verify  MASH  Off  Net  6 


28.  COMMD  Calls  in  Connect 
with  Div  COR  DIV/ARTY 
Tells  DIV/ARTY  to  put 
Div  CMD  on  Net  3 


29.  MON  link  between  CQMHO  6 
DIV  CMD 


30.  Station  6 calls  in 
Put  me  on  FOX  1 RADIO 
and  Ring  me  on  LCL 


3i . Ring  up  Station  6 


32.  Connect  Station  6 to 
FOX  1 Net 


33.  Verify  Net  4 
Net  4 


be  put  on  Net  8:  FI,  BN, 
CMD,  DIV/ARTY,  S2.  COMMO. 
MASH,  MESS  Security  and 
Station  6 and  DDT  4 


35.  Power  Down 

Reconfigure  Nets 

Net  1 

Net  2 

Net  3 

Net  4 

Net  5 

Net  6 

Net  7 

Net  8 


ACCURACY 


ERROR  DESCRIPTION 


ANS 

NET,  1.  DDT, 
NET.  6,  DDT, 

1,  DISC.  DOT,  6.  CONN 

1,  CONN 

WIRE  1,2  RAD 

1,5,9  DDT  6 

WIRE  8,32, 

RAD  3 DDT  1 

ANS 

NET,  6,  WIRE. 

8.  DISC 

WIRE  32  RAO 

3 DDT  1 

WIRE  --  8 

1 

ANS 

WIRE,  1.  RING 

Met  3,  MON  Cannot  monitor  a link 

ANS 

ANS 

■Changes  CTB  Connections 

Puts  Station  { 

S on  LCL  14 

LCL,  14.  RING 

NET,  4.  LCL. 

14.  CONN, 

LCL,  15.  DISC 

DOT  4 LCL 

6,7,14 

NET.  8.  RAO, 

1,  CONN, 

RAD,  5,  CONN. 

WIRE.  1,  CONN 

WIRE,  4,  CONN 

. WIRE.  7,  CONN. 

WIRE,  S.  CONN 

, LCL,  2,  CONN 

LCL.  5.  CONN. 

LCL  14,  CONN. 

DDT,  4,  CONN 

RAD 

9 

WIRE 

2 

DDT  6 

RAD 

10 

DDT 

2 LCL  18, 

.19 

RAO 

4 

DOT 

3 

LCL 

6, 

,7 

RAO 

2 

WIRE 

30. 

31  DOT 

5 LCL  3,4,8.16.17 

RAO 

3 

WIRE 

32 

DOT  1 

RAD 

6, 

.7.8 

WIRE 

3.5.( 

S 

RAO 

1, 

,5  WIRE  1 

.4.7,8 

LCL  2, 

5,14 

DDT 

4 

36.  Verify  Nets 


APPENDIX  3 

SAMPLE  TEST  PARTICIPANT  QUESTIONNAIRE 
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PARTICIPANT  QUESTIONNAIRE 

1.  PERSONAL  DATA 

Participant  Number  Date  

Age  Height  Weight  

Years  in  Service  Grade 

Time  in  Grade  Duty  Assignment  

Weeks  of  Experience  in  Duty  Assignment  

Visual  Acuity  AGCT  Score  

Any  Physical  Disabilities  If  Yes  Indicate  Type  

2.  EXPERIENCE 

Schooling  (Circle  highest  year  completed) 

Grade  School  Jr.  High  School  High  School  College 
123456  789  10  11  12  123456 

Service  Training: 

List  all  service  schools  attended  and  months  in  each. 


Weeks  of  Experience  Operating  CCU 


3.  ANTHROPOMETRIC  DATA  (Sitting) 

(15)  Sitting  Height (17)  Eye  Height 

(24)  Elbow  Resting  Height 

Horizontal  Arm  Reach (14)  Vertical  Arm  Reach 

(23)  Elbow-Fingertip  Reach (31)  Buttock-Heel  Length 

4.  UNUSUAL  CLOTHING  (other  than  fatigues) 


167 


1.  In  what  general  ways  could  the  CCU  Training  program  have  better 
prepared  you  for  your  present  task? 

More  practice  on  real  equipment 

Better  simulation  of  real  equipment 

More  extensive  theoretical  background 

Mo.  e extensive  operational  lectures 

More  detail  on  all  aspects  of  training 

Other,  specify; 

2.  What  specific  aspect  of  your  task  could  be  improved  by  additional 
training? 

3.  Did  you  receive  any  on-the-job  training  once  assigned  to  CCU? 

4.  What  was  lacking  in  your  on-the-job  training  once  assigned  to  CCU? 

Too  few  training  personnel 

Poor  system  input  simulation 

Inadequate  practice  time 

Inadequate  equipment  familiarization  time 

5.  How  many  hours  of  practice  do  you  get  each  week?  

6.  Does  the  amount  of  practice  you  get  allow  you  to  perform  at  an 

adequate  level  during  peak  loads  and/or  during  emergency?  

7.  During  practice  sessions  do  you  have  realistic  inputs  to  deal  with? 
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8. 


You  have  been  trained  (formally  or  otherwise)  to  perform  a particular 
sequence  of  tasks  in  order  to  operate  your  console.  In  what  ways  do 
you  deviate  from  this  sequence  in  order  to  perform  more  effectively? 


9.  Do  you  find  the  task  of  operating  the  CCU  uninteresting  and  dull?  

10.  Would  you  rather  operate  a different  console?  

11.  While  working  at  your  console,  when  do  you  begin  to  feel  bored? 

(minutes).  How  about  fatigued?  (minutes).  When 

do  you  think  your  "being  tired"  began  to  affect  your  efficiency  on  the 
equipment?  

12.  Do  you  ever  have  difficulty  knowing  when  to  start  your  task?  

If  yes,  please  explain  the  problem. 


13.  If  given  the  opportunity,  in  what  ways  would  you  change  the  placement 
of  certain  console  displays,  knobs,  dials,  etc.? 


14.  If  given  the  opportunity  to  change  the  placement  of  the  console,  where 
would  you  put  it  relative  to  other  operational  equipment? 


I 
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15.  Please  list  any  display  interpretation,  dial  or  indicator  reading 
problems,  or  control  utilization  problems  which  you  frequently  have 
in  performing  your  task. 


Cause  of  Problem  (e.g., 
placement,  scale  info., 
rate,  knobs  too  close. 

Display  or  Control  Problem  etc.) 


16.  Listed  below  are  some  factors  which  might  cause  an  operator  to  perform 
at  a level  lower  than  his  best.  Check  those  which  cause  you  difficulty 
and  explain  the  problem  it  causes. 

Environment  too  dark  

Environment  too  bright  

Too  much  glare  

Environment  too  cool  

Environment  too  warm  

Environment  too  noisy  

Environment  too  crowded  

Seats  uncomfortable  

Not  enough  fresh  air  

Console  lighting  too  low  

Console  lighting  too  bright  

Others  (please  specify)  
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17.  What  mechanical  aspects  of  the  equipment  did  you  have  difficulty  with 
during  operation? 


a.  Activating  switches  f 

b.  Discerning  information  on  display  g 

c.  Writing  on  forms  h 

d.  Reaching  equipment  i 

e.  Access  to  operating  position  j 


18.  Which  task  or  sequence  of  operations  did  you  have  difficulty  in 
performing? 


a. 

Start  up 

e.  Net  changes 

b. 

CTB  connections 

f. 

c. 

Net  set  up 

g- 

d. 

Card  replacement  maintenance 

h. 

What  do  you  think  the  r-eason  for  this  difficulty  is? 

a.  Layout  of  the  equipment 

b.  Infrequency  of  such  operations 

c.  Complexity  of  such  operations 

d.  Inadequacy  of  instruction 

e.  Other 

19.  Were  you  able  to  maintain  a comfortable  position  during  operation? 

a.  yes  b.  no 

20.  Did  you  have  to  strain  yourself  frequently  to  reach  all  the  controls? 

a.  yes  b.  no 
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21.  What,  in  your  opinion,  should  be  the  maximum  length  of  time  an 
operator  spends  on  duty? 


a. 

2 

hrs 

d. 

8 hrs 

b. 

4 

hrs 

e. 

10  hrs 

c. 

6 

hrs 

f. 

12  hrs 

22.  Has  the  design  of  any  TACFIRE  equipment  hindered  your  ability  to 
operate  (or  to  perform  maintenance)  effectively? 

23.  Has  the  design  of  any  TACFIRE  equipment  caused  you  to  make  an  error 
or  misinterpret  an  output  during  operation  of  the  system? 

24.  Have  any  environmental  conditions  hindered  efficient  operation  and 
maintenance? 

25.  Have  noise  levels  interfered  with  normal  operations  at  any  time? 

26.  Has  there  been  any  damage  to  equipment  or  loss  of  data  as  a result  of 
improper  actions  or  improper  interconnections  of  components? 

27.  What  criticisms  do  you  have  of  the  equipment  layout,  and  what 
suggestions  do  you  have  to  improve  your  effectiveness  as  an  operator? 
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APPENDIX  4 


SAMPLE  HFE  CHECKLIST 
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HUMAN  FACTORS  CHECKLIST  — OPERATION 


1.  Are  noise  levels  generated  by  components  interfering 
with  personnel  speaking  at  conversational  levels? 

2.  Is  adequate  space  provided  for  each  operator  in 
front  of  his  respective  equipment? 

3.  Can  each  operator  leave  his  working  position  and 
the  compartment  without  disturbing  any  other 
operator? 

4.  From  their  operating  positions,  can  supervisors 
observe  all  the  personnel  in  the  compartment  under 
their  charge? 

5.  Are  ladders,  climbing  rings,  hand-holds,  and  rails, 
walkways,  etc.,  present  where  needed  and  are  they 
adequate  enough  to  provide  sure  footing  and 
gripping  even  when  icy  or  highly  waxed? 

6.  Are  mechanical  and  electrical  interlocks  provided 
to  prevent  turning  equipment  on,  or  moving  it,  when 
men  are  in  positions  which  would  be  dangerous? 

7.  Is  writing  space  provided  where  tasks  involve  the 
use  of  books,  manuals,  or  forms? 

8.  Is  the  CRT  display  legible  during  all  presentations? 

9.  Do  all  of  the  operators  requiring  information  from 
a common  display  have  a clear  line  of  sight  from 
their  operating  positions? 

10.  Is  the  major  display  for  each  operator  mounted 
perpendicular  to  his  normal  line  of  sight? 

11.  Are  primary  controls  and  displays  placed  within 
the  optimal  visual  and  manual  spaces  on  the 
console  or  unit? 

12.  Can  a display  be  read  easily  from  the  expected  or 
normal  locations  of  all  operators  who  require  the 
information? 

13.  Is  parallax  on  the  display  minimized  for  the 
operator's  normal  visual  axis? 

14.  Is  there  visual  flicker  on  any  display? 
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YES  NO  N/A 


15.  Does  activating  a control  obscure  a visual  display 
or  control  markings? 

16.  Does  any  equipment  design  cause  operator  discomfort, 
including  stress  and  fatigue? 

17.  Are  emergency  controls  and  displays  placed  in 
readily  accessible  positions? 

18.  Are  emergency  controls  quickly  identifiable  for 
maximum  speed  of  operation? 

19.  If  emergency  controls  affect  the  equipment 
extensively  when  operated,  does  it  require  at 
least  two  distinct  motions  to  operate  them? 

20.  Are  displays  that  must  be  check-read  grouped 
together? 

21.  Does  every  control  and  display  on  the  panel  have 
a descriptive  legend  associated  with  it? 

22.  Is  there  adequate  separation  between  controls,  so 
that  they  can  be  operated  easily  without 
accidentally  disturbing  adjacent  controls? 

23.  When  controls  must  be  adjusted  as  an  operator 
observes  a display,  can  he  reach  and  adjust  them 
easily  while  viewing  the  visual  display? 

24.  Are  successive  control  movements  interrelated 
(i.e.,  does  one  movement  pass  easily  into  the  next)? 

25.  Do  controls  used  in  rapid  sequence  have  uniform 
direction  of  motion? 

26.  Are  control  movements  consistent  for  all  of  the 
equipment  which  one  operator  uses? 

27.  Does  the  method  of  preventing  accidental  activation 
of  any  control  increase  the  time  to  operate  the 
control  to  such  an  extent  that  it  is  unacceptable? 

28.  If  an  operator's  task  is  complex,  are  the  controls 
distributed  so  that  no  one  limb  is  overburdened? 

29.  Are  controls  associated  with  similar  functions 
located  in  the  same  relative  position  from  panel 
tfo  panel  ? 
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YES  NO  ^ 


30.  Are  controls  of  the  same  size  used  for  performing 
the  same  functions  on  different  equipment? 

31.  Is  the  operator  warned  of  a failure  in  the  unit? 

32.  Is  overall  display  area  as  small  as  possible, 
consistent  with  legibility  at  the  required  reading 
distance? 

33.  When  displays  are  used  sequentially,  are  they 
aligned  horizontally  from  left  to  right,  and  as 
close  to  each  other  as  possible? 

34.  When  the  operator  must  monitor  or  perform  a sequence 
of  operations,  are  the  displays  arranged  in  the 
actual  order  of  events? 

35.  Are  the  most  important  controls  located  in  front 
of  the  operator  within  easy  reach,  and  between 
elbow  and  shoulder  height? 

36.  Can  the  shapes  of  controls  be  discerned  both 
visually  and  by  touch  when  applicable? 

37.  Do  color  controls  provide  ample  contrast  with 
the  background? 

38.  Is  color  coding  consistent  with  established 
standards  of  the  system? 

39.  Is  color  coding  limited  to  the  following  six 
colors:  white,  black,  red,  yellow,  green  and 
bl  ue? 

40.  Is  information  presented  in  the  most  immediately 
meaningful  form  (no  interpretation  or  decoding 
required)? 

41.  Is  information  displayed  to  the  accuracy  required 
for  the  operator's  decisions  or  actions,  and 
preferably  no  more  accurately  than  required? 

42.  Is  information  for  different  types  of  activities 
(e.g.,  operation  and  maintenance)  combined  when 
not  necessary? 
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APPENDIX  5 

SAMPLE  TEST  SCENARIO 
(CCU  Test  Segment  4) 
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1 

TASK 

i 

FDO 

(Tst  Dir) 

BN  CDR  S2  COMM  0. 

DIV/ARTY  DIV  CDR 

R,ADI0  1 M.A.S.H.  MESS 

SECURITY  STA.  6 

1 

Answer  Answer  Answer 

Answer  ' 

Answer  Answer  Answer 

Answer  Answer 

2 

Verify 
nets  and 
report  to 
ACC  opr. 

3 

"Link  me 

Security" 

(1) 

Answer  & 
ring  off 
after  talk 
(2) 

4 

1 

"Put  me 
on  CF 
neu" 

5 

Verify 
nets  1 8. 

4, 

update 
net  form 

6 

■ 

Answer 

(3) 

Answer 

(2) 

7 

"Link  me 
to  Sta- 
tion 6" 

(1) 

Answer, 

then 

ring  off 
(2) 

8 

Monitor 
the  link 

Answer 

(3) 

Answer 

(2) 

"Put  me 
back  on 
link  with 
Comm  0 & 
Radio  1" 
(1) 

9 

1 

Ring  on 
net.  Ask 
for  time 
of  day 

"Put  me 
on  Bgd 

FSO  net" 

11 

Verify 
nets  1 
& 3. 

Update 

form 

1 

12 

Monitor 
all  nets 

Answer 

Div/Arty 

(3) 

Talk  on 

FSO  net 
(2) 

13 

Tell  Link 

1 that 
you  wi 1 1 
dis- 
connect 
them. Dis- 
connect 
Link  1 
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TASK 

FDO 

(Tst  Dir) 

BN  COR  S2  COHf  0. 

DIV/ARTY  DIV  CDR 

RADIO  1 M.A.S.H.  MESS 

SECURITY  STA.  6 

14 

Power 

Off.  "Re- 

config. 

nets 

15 

"Add  Sta. 
9 to  LBU 
net" 

16 

Veri fy 
net  2 
& update 
net  form 

17 

"Link  me 
to 

M.A.S.H." 

(I) 

Answer 

"No 

plasma , 
call  Sta. 

6"  Ring 
off. 

(?) 

18 

"Link  me 
to  Sta. 

6" 

(1) 

Answer, 
don ' t 
ring  off 
(2) 

19 

"Where's 

Div/Arty? 

Put  him 
back  on 

CF  net" 

Answer 

20 

"Exchange 
my  DDT 
with  Fox 

3 net" 

21 

Veri fy 
nets  1 

A 6. 

update 

form 

22 

■n 

"Dis- 
connect 
me  from 
my  net" 

23 

Verify 
net  6 

A 9, 

update 

form 

" Connect 
me  with 
Oiv  CDR 
at 

Oiv/Arty" 

(1) 

Set  up  Answer 

link  with 

BN  & Oiv. 

Cdr 

(?) 

25 

"This  is 

Sta  6 
calling 
from  Rad  1 

Put  Sta  6 
on  a new 
local 
connec- 
tion. " 
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